kolu
Docs

tutorial

Make it a daemon.

In Your first surface, fleet-top died when you closed the terminal. Here we make it durable: one long-lived process that holds a single-instance gate, serves the same surface over a unix socket, and can be replaced by a fresh build while a client stays connected. The surface itself does not change at all — only how it is reached and kept alive.

Wrap the engine in a daemon

daemonMain is the entire gate → serve → teardown lifecycle. It claims an atomic single-instance gate, serves the router over a socket, waits for the lifetime to end, then closes the socket and releases the gate — returning a DaemonExit rather than calling process.exit, so the whole thing is drivable from a test.

/**
 * The daemon — part 1's `top`, now single-instance and durable.
 *
 * `daemonMain` is the whole `gate → serve → teardown` skeleton:
 *
 *   acquirePidGate(GATE_PATH)     — atomic single-instance claim (link(2)).
 *        │ held? → exit 0 (a live daemon already serves this scope)
 *        ▼
 *   serve top.router over SOCKET  — the @kolu/surface unix-socket listener
 *        ▼
 *   wait for lifetime to end      — { kind: "forever" }: only a signal / abort
 *        ▼
 *   close socket · release gate · return DaemonExit
 *
 * We pick `forever` — an idle `top` still watches your machine, so an idle
 * timeout would wrongly kill it. `daemonMain` never calls `process.exit`; it
 * RETURNS a `DaemonExit` the bin maps to a code (`daemonExitCode`), which is
 * what makes the whole lifecycle drivable in-process from a test.
 *
 * The same flattened `createTop()` router from part 1 is served verbatim — the
 * daemon changes how it's *reached* (a durable socket instead of a fresh
 * per-connection process), not what it serves.
 */

import { daemonExitCode, daemonMain, stderrLogger } from "@kolu/surface-daemon";
import { GATE_PATH, SOCKET_PATH } from "../common/paths";
import { createTop } from "./top";

async function main(): Promise<void> {
  const top = createTop();
  top.start();

  const exit = await daemonMain({
    gatePath: GATE_PATH,
    socketPath: SOCKET_PATH,
    router: top.router,
    lifetime: { kind: "forever" },
    log: stderrLogger(),
    onReady: ({ socketPath, pid }) =>
      process.stderr.write(
        `fleet-top daemon listening on ${socketPath} (pid ${pid})\n`,
      ),
  });

  top.dispose();
  process.exit(daemonExitCode(exit));
}

main().catch((err) => {
  process.stderr.write(`daemon fatal: ${(err as Error).message}\n`);
  process.exit(1);
});

We choose { kind: "forever" }: an idle top still watches your machine, so an idle timeout would wrongly kill it. Note the router is the same createTop() from part 1, served verbatim.

  1. Start the daemon.

    pnpm run daemon
    fleet-top daemon listening on /run/user/1000/fleet-top.sock (pid 48213)

    That friendly line comes from onReady; stderrLogger also writes a structured JSON line per event to stderr, so you will see a {…"msg":"daemon listening"…} line alongside it. Leave this running.

  2. In a second terminal, start it again.

    pnpm run daemon

    It exits immediately, logging one line that a live daemon already holds the gate (daemon already running; yielding to the live instance), and steps aside. That is the single-instance guarantee: acquirePidGate is an atomic link(2) claim.

Supervise it

A daemon you cannot spawn, watch, and replace is not much use. createEndpoint is the supervisor half of the spine — it runs in the client, never the daemon. ensure() is an always-recycle boot: kill any live survivor, wait for its pid to be reaped, spawn a fresh daemon so it outlives us, connect, and handshake. Every transition is reported.

/**
 * The supervisor — spawn, watch, and LIVE-recycle the daemon.
 *
 * `createEndpoint` is the supervisor half of the spine (it runs in the CLIENT,
 * never the daemon). It takes the daemon from nothing to a live, handshaken
 * connection and reports every transition:
 *
 *   connecting → connected      (recycled, socket up, handshake passed)
 *   connecting → dead           (couldn't spawn / connect)
 *   connected  → degraded       (the daemon died mid-session)
 *
 * `ensure()` is the always-recycle boot: a live survivor is killed, then a
 * fresh daemon is spawned — every boot exercises kill → `waitForPidGone` →
 * spawn → connect (composed from `@kolu/surface-daemon`'s gate primitives).
 * `survivableSpawnDriver` launches the daemon so it OUTLIVES us (systemd-run
 * --user under a service; detached + unref otherwise).
 *
 * The finale is the LIVE recycle under a connected client: `restart` runs the
 * fixed `capture → drain → recycle → reattach` sequence. This demo makes no
 * survival promise, so it supplies the degenerate steps (B2's boot recycle);
 * part 3's remote fan-out is where the same sequence carries real per-host
 * session state. The client we hold reconnects on the far side of the recycle.
 */

import { fileURLToPath } from "node:url";
import { stderrLogger } from "@kolu/surface-daemon";
import {
  createEndpoint,
  restart,
  survivableSpawnDriver,
} from "@kolu/surface-daemon-supervisor";
import { GATE_PATH, SOCKET_PATH } from "../common/paths";
import { connectTop, type TopClient, type TopIdentity } from "./connect";

async function firstFrame<T>(
  source: AsyncIterable<T> | Promise<AsyncIterable<T>>,
): Promise<T> {
  for await (const frame of await source) return frame;
  throw new Error("stream closed before its snapshot frame");
}

async function main(): Promise<void> {
  const log = stderrLogger();

  // The daemon binary the driver spawns. In a Nix build this is the realised
  // executable; from source we point `node` at the daemon entry through tsx.
  const daemonEntry = fileURLToPath(
    new URL("../daemon/main.ts", import.meta.url),
  );

  const endpoint = createEndpoint<TopClient, TopIdentity>({
    hostId: "local",
    gatePath: GATE_PATH,
    socketPath: SOCKET_PATH,
    driver: survivableSpawnDriver({
      binPath: process.execPath, // node
      args: ["--import", "tsx/esm", daemonEntry],
      env: {
        FLEET_TOP_GATE: GATE_PATH,
        FLEET_TOP_SOCKET: SOCKET_PATH,
      },
      unitPrefix: "fleet-top",
      // Launched from source (tsx), not a built binary — so the driver forces
      // the detached branch even under a systemd session.
      fromSource: true,
    }),
    connect: () => connectTop(SOCKET_PATH),
    log,
    onStatus: (hostId, status) =>
      process.stderr.write(`[supervisor] ${hostId}: ${status.state}\n`),
  });

  // Boot: always-recycle → spawn → connect. Throws (after reporting `dead`) if
  // it cannot bring the daemon up.
  await endpoint.ensure();

  const conn = endpoint.current();
  if (conn === undefined)
    throw new Error("endpoint connected but current() is undefined");
  const mem = await firstFrame(conn.client.surface.memory.get({}));
  process.stderr.write(
    `[supervisor] connected — daemon reports ${conn.identity.cores} cores, ` +
      `${(mem.used / 1e9).toFixed(1)} GB used\n`,
  );

  // The LIVE recycle: kill the daemon under us and stand a fresh one up, with
  // the status held at one honest "restarting". Degenerate steps — nothing to
  // preserve in this part.
  await restart(endpoint, {
    capture: async () => undefined,
    drain: async () => {},
    reattach: async () => {},
  });
  process.stderr.write("[supervisor] live recycle complete\n");

  endpoint.current()?.dispose();
}

main().catch((err) => {
  process.stderr.write(`supervisor fatal: ${(err as Error).message}\n`);
  process.exit(1);
});

The finale is the payoff: restart runs the fixed capture → drain → recycle → reattach sequence — a live recycle. The daemon is killed and a fresh one stood up underneath, while the connection we hold reconnects on the far side. This demo has nothing to preserve, so it passes the degenerate steps; part 3 is where the same sequence carries real per-host state.

  1. With a daemon still running from before, start the supervisor.

    pnpm run supervisor
    [supervisor] local: connecting
    [supervisor] local: connected
    [supervisor] connected — daemon reports 8 cores, 9.2 GB used
    [supervisor] local: connecting
    [supervisor] local: connected
    [supervisor] live recycle complete

    The second connecting / connected pair is the live recycle itself: restart re-runs ensure(), so the endpoint reconnects to the fresh daemon and re-fires its status. (stderrLogger also emits a structured JSON line per event to stderr; the [supervisor]-prefixed lines above are the ones to read.)

Watch the order: the supervisor recycled the daemon you had running (the always-recycle boot), connected to the fresh one, read a cell straight off it, then recycled it again live — and the client it held simply reconnected. No dropped session, no lost socket.

Next

fleet-top now survives on one machine. In Across the hosts you will reach a fleet-top running on another box over ssh — and watch the same surface arrive in your browser from a machine you never deployed to by hand.