kolu
Docs

operate

Stop it cleanly.
Test it beside itself.

Two procedures here run kolu next to, or instead of, the kolu already on this machine: stopping the stack by hand (what a rollback needs), and launching an isolated second instance. Both act on the daemons, padi and kaval, not the web server, which you can restart freely.

Before you start

  • There are two state directories. KOLU_STATE_DIR is the server’s config store (~/.config/kolu by default). KOLU_PADI_STATE_DIR is padi’s state root (~/.local/state/padi): saved sessions, padi.log, and the anchor for everything else. The production launcher supplies both. A bare padi with neither set refuses to start rather than guess.
  • The padi state root is an identity. Its absolute path is hashed to a short digest, and that digest names every socket and pid gate the daemons use. Two kolus with different padi state roots compute different paths, so neither can see the other’s daemons.
  • Each daemon claims a pid gate beside its socket, in the per-user runtime directory.
FileHolder
<runtime>/padi-<digest>/padi.pidthe padi daemon
<runtime>/kaval-<digest>/kaval.pidthe kaval daemon under it
<runtime>/kaval-<digest>/state-rootthe state-root path that digest stands for

A gate file is one line: the pid, a tab, and the process start time. So cut -f1 gets you the pid. Runtime directories are wiped on reboot; the state roots persist.

<runtime> is $XDG_RUNTIME_DIR where it is set, which on Linux is /run/user/$UID. Where it is not set (macOS), the daemons fall back to /tmp and carry your uid in the directory name instead, as /tmp/padi-<digest>-$UID/. The commands below spell the Linux paths; on macOS substitute that shape.

Where the logs are

LogWhereWhat only it can tell you
the server’sits stdout, which is the journal under systemdwhy a host’s daemon was stopped or replaced: which policy fired, and the evidence behind it
padi.log.Nthe padi state root, on that hostwhat one host’s daemon did: sessions, terminals, its own view of kaval
padi.stderr.logbeside itcrashes pino cannot see: native stderr, uncaught stacks

The daemons write their own files, because they are spawned detached. Whoever launched the server owns its output instead: the journal under systemd, your terminal under a bare kolu web, where it is lost when the terminal closes. Redirect it if you leave one running:

kolu web > ~/kolu-server.log 2>&1

The first row is the one people go looking for after a surprise. The decision to stop or replace a padi is the server’s: padi’s own log records only that it was asked to stop.

The daemon logs are numbered, and roll by size. Current lines are in padi.log.1, older rounds in .2 and up. The un-numbered padi.log stays empty and is not where the lines go.

Each config.json also keeps a backup ring: a backups/ directory beside the file, holding the ten most recent boot-time and daily snapshots. The server’s snapshot carries your preferences, the host pool and the Conductor’s pieces of work and memory. It does not carry the Conductor’s conversation or the files a finished work kept, which are copies on disk rather than state in this file. To restore one, see Troubleshooting.

Stop the stack

  1. Stop the service first. Otherwise its supervisor restarts the web server, which spawns fresh daemons the moment you kill the ones you meant to stop.

    systemctl --user stop kolu

    On macOS the home-manager module runs a launchd LaunchAgent instead; unload that. If you started kolu by hand, stop that process.

  2. SIGTERM each daemon by the pid in its gate file.

    kill -TERM "$(cut -f1 /run/user/$UID/padi-*/padi.pid)"
    kill -TERM "$(cut -f1 /run/user/$UID/kaval-*/kaval.pid)"
  3. Verify they are gone before starting anything in their place.

    ps -p <pids>

The whole shutdown as one line, service and both daemons, pids read from the gates:

systemctl --user stop kolu; kill -TERM $(cut -f1 /run/user/$UID/padi-*/padi.pid /run/user/$UID/kaval-*/kaval.pid 2>/dev/null) 2>/dev/null

padi sweeps dead gate directories at every start, its own and its kaval’s, under the runtime directory and under /tmp. It removes only the ones whose gate file names a process that is gone, so the glob may still hand you a pid that no longer exists. Signalling a dead pid is a harmless error, which the redirections absorb. Check with ps rather than trusting the exit status.

Send SIGTERM, not SIGKILL. padi’s last act on that signal is to persist the live session. Its continuous autosave is throttled, so the blob on disk otherwise trails the live workspace by up to half a second, and which terminal is active by up to five.

SIGKILL skips that final capture and makes the last autosave your recovery point. Keep SIGKILL for a daemon that has already ignored a SIGTERM, or one you observed stopped, since a stopped process cannot act on SIGTERM at all.

Roll back to an older release

Upgrading is hands-off. When a release changes the protocol the daemons speak, the new supervisor probes whatever holds the socket, recognizes a previous-epoch daemon, corroborates it against the gate file and the OS process table, and takes it over. That happens on this machine and on each remote host, with nothing to run by hand. The story users see is in Troubleshooting.

A process holding the socket that the supervisor cannot prove is one of its own daemons is never signalled. It is refused, and the refusal names the holder’s pid and command, so you can deal with it by hand. If something holds kaval’s socket, accepts connections and then says nothing, padi takes about 90 seconds to refuse it: three tries, each waiting out a 30-second handshake deadline.

Going the other way is manual, because the takeover logic exists only in the newer binary. An older kolu meets the new daemon’s socket, fails to speak its protocol, and refuses, so you clear the way yourself.

  1. Stop the stack with the procedure above, SIGTERM and all. The final capture it triggers is what the older build restores from.

  2. Confirm nothing survived — ps -p on the pids you signalled. A daemon still holding its socket makes the older build refuse exactly as before.

  3. Start the older version. It spawns fresh daemons of its own generation and restores your terminals from the saved session blobs.

Run a second kolu beside a live one

A second kolu on this box shares its RAM and its kernel with production, so an ephemeral box is the safer answer. The settings below are the floor of isolation, not the ceiling. Three of them make a nix run instance its own thing:

export KOLU_STATE_DIR="$HOME/tmp/kolu-test/state"     # server config store (default: ~/.config/kolu)
export KOLU_PADI_STATE_DIR="$HOME/tmp/kolu-test/padi" # padi state root: sessions, gates, sockets
nix run github:juspay/kolu -- web --port 8681             # production listens on 7681

The launcher honours both variables when they are already set, and falls back to the production paths when they are not. That is why forgetting one is dangerous rather than merely untidy.

A different --port isolates nothing by itself. padi and kaval are keyed by the digest of the padi state root, not by the listen port, so --port moves the web listener and leaves the daemons where they were. It lets two servers coexist; it does not keep them apart.

To check your work, ask each running kaval which state root it belongs to:

cat /run/user/$UID/kaval-*/state-root

Two different paths mean the two instances are genuinely separate. Your test path appearing alone, or ~/.local/state/padi appearing twice, means they are not.