operate
Stop it cleanly.
Test it beside itself.
Two procedures here run kolu next to, or instead of, the kolu already on this
machine: stopping the stack by hand (what a rollback needs), and launching an
isolated second instance. Both act on the daemons, padi and
kaval, not the web server, which you can restart freely.
Before you start
- There are two state directories.
KOLU_STATE_DIRis the server’s config store (~/.config/koluby default).KOLU_PADI_STATE_DIRis padi’s state root (~/.local/state/padi): saved sessions,padi.log, and the anchor for everything else. The production launcher supplies both. A barepadiwith neither set refuses to start rather than guess. - The padi state root is an identity. Its absolute path is hashed to a short digest, and that digest names every socket and pid gate the daemons use. Two kolus with different padi state roots compute different paths, so neither can see the other’s daemons.
- Each daemon claims a pid gate beside its socket, in the per-user runtime directory.
| File | Holder |
|---|---|
<runtime>/padi-<digest>/padi.pid | the padi daemon |
<runtime>/kaval-<digest>/kaval.pid | the kaval daemon under it |
<runtime>/kaval-<digest>/state-root | the state-root path that digest stands for |
A gate file is one line: the pid, a tab, and the process start time. So
cut -f1 gets you the pid. Runtime directories are wiped on reboot; the state
roots persist.
<runtime> is $XDG_RUNTIME_DIR where it is set, which on Linux is
/run/user/$UID. Where it is not set (macOS), the daemons fall back to /tmp
and carry your uid in the directory name instead, as /tmp/padi-<digest>-$UID/.
The commands below spell the Linux paths; on macOS substitute that shape.
Where the logs are
| Log | Where | What only it can tell you |
|---|---|---|
| the server’s | its stdout, which is the journal under systemd | why a host’s daemon was stopped or replaced: which policy fired, and the evidence behind it |
padi.log.N | the padi state root, on that host | what one host’s daemon did: sessions, terminals, its own view of kaval |
padi.stderr.log | beside it | crashes pino cannot see: native stderr, uncaught stacks |
The daemons write their own files, because they are spawned detached. Whoever
launched the server owns its output instead: the journal under systemd, your
terminal under a bare kolu web, where it is lost when the terminal closes.
Redirect it if you leave one running:
kolu web > ~/kolu-server.log 2>&1
The first row is the one people go looking for after a surprise. The decision to stop or replace a padi is the server’s: padi’s own log records only that it was asked to stop.
The daemon logs are numbered, and roll by size. Current lines are in
padi.log.1, older rounds in .2 and up. The un-numbered padi.log stays
empty and is not where the lines go.
Each config.json also keeps a backup ring: a backups/ directory beside the
file, holding the ten most recent boot-time and daily snapshots. The server’s
snapshot carries your preferences, the host pool and the
Conductor’s pieces of work and memory. It does not
carry the Conductor’s conversation or the files a finished work kept, which are
copies on disk rather than state in this file. To restore one, see
Troubleshooting.
Stop the stack
-
Stop the service first. Otherwise its supervisor restarts the web server, which spawns fresh daemons the moment you kill the ones you meant to stop.
systemctl --user stop koluOn macOS the home-manager module runs a launchd LaunchAgent instead; unload that. If you started kolu by hand, stop that process.
-
SIGTERM each daemon by the pid in its gate file.
kill -TERM "$(cut -f1 /run/user/$UID/padi-*/padi.pid)" kill -TERM "$(cut -f1 /run/user/$UID/kaval-*/kaval.pid)" -
Verify they are gone before starting anything in their place.
ps -p <pids>
The whole shutdown as one line, service and both daemons, pids read from the gates:
systemctl --user stop kolu; kill -TERM $(cut -f1 /run/user/$UID/padi-*/padi.pid /run/user/$UID/kaval-*/kaval.pid 2>/dev/null) 2>/dev/null
padi sweeps dead gate directories at every start, its own and its kaval’s,
under the runtime directory and under /tmp. It removes only the ones whose
gate file names a process that is gone, so the glob may still hand you a pid
that no longer exists. Signalling a dead pid is a harmless error, which the
redirections absorb. Check with ps rather than trusting the exit status.
Send SIGTERM, not SIGKILL. padi’s last act on that signal is to persist the live session. Its continuous autosave is throttled, so the blob on disk otherwise trails the live workspace by up to half a second, and which terminal is active by up to five.
SIGKILL skips that final capture and makes the last autosave your recovery point. Keep SIGKILL for a daemon that has already ignored a SIGTERM, or one you observed stopped, since a stopped process cannot act on SIGTERM at all.
Roll back to an older release
Upgrading is hands-off. When a release changes the protocol the daemons speak, the new supervisor probes whatever holds the socket, recognizes a previous-epoch daemon, corroborates it against the gate file and the OS process table, and takes it over. That happens on this machine and on each remote host, with nothing to run by hand. The story users see is in Troubleshooting.
A process holding the socket that the supervisor cannot prove is one of its own daemons is never signalled. It is refused, and the refusal names the holder’s pid and command, so you can deal with it by hand. If something holds kaval’s socket, accepts connections and then says nothing, padi takes about 90 seconds to refuse it: three tries, each waiting out a 30-second handshake deadline.
Going the other way is manual, because the takeover logic exists only in the newer binary. An older kolu meets the new daemon’s socket, fails to speak its protocol, and refuses, so you clear the way yourself.
-
Stop the stack with the procedure above, SIGTERM and all. The final capture it triggers is what the older build restores from.
-
Confirm nothing survived —
ps -pon the pids you signalled. A daemon still holding its socket makes the older build refuse exactly as before. -
Start the older version. It spawns fresh daemons of its own generation and restores your terminals from the saved session blobs.
Run a second kolu beside a live one
A second kolu on this box shares its RAM and its kernel with production, so an
ephemeral box is the safer answer. The settings below are the floor of
isolation, not the ceiling. Three of them make a nix run instance its own
thing:
export KOLU_STATE_DIR="$HOME/tmp/kolu-test/state" # server config store (default: ~/.config/kolu)
export KOLU_PADI_STATE_DIR="$HOME/tmp/kolu-test/padi" # padi state root: sessions, gates, sockets
nix run github:juspay/kolu -- web --port 8681 # production listens on 7681
The launcher honours both variables when they are already set, and falls back to the production paths when they are not. That is why forgetting one is dangerous rather than merely untidy.
A different --port isolates nothing by itself. padi and kaval are keyed by
the digest of the padi state root, not by the listen port, so --port moves the
web listener and leaves the daemons where they were. It lets two servers
coexist; it does not keep them apart.
To check your work, ask each running kaval which state root it belongs to:
cat /run/user/$UID/kaval-*/state-root
Two different paths mean the two instances are genuinely separate. Your test
path appearing alone, or ~/.local/state/padi appearing twice, means they are
not.