Seven weeks to a remote terminal

Here’s a feature that sounds like an afternoon of work. You already SSH into a build box every day. You’d like a terminal in Kolu — the terminal-native cockpit I build coding agents in — to run its shell on that box instead of your laptop. Point Kolu at the host, get a shell. How hard can it be?

I filed that as issue #951 on May 21. I closed it on July 14, the day I wrote this. In between were seven weeks, roughly two dozen pull requests I built and then threw in the trash, a 6,000-line PR nobody could review, and one afternoon where a daemon I’d written deleted a live 20-terminal session out from under me. The answer, when it finally came, was almost boring: one daemon per host, and you switch between hosts instead of mixing them. Getting there was not boring at all.

This is the story of the wrong turns. Not because the destination is interesting — it’s a paragraph — but because the seven weeks are the actual work, and nobody writes that part down.

The ask: a remote shell that isn’t special

The trap in issue #951 is the word “exactly.” Running a shell over SSH is the easy part — ssh -tt host and you’re most of the way there. The hard part is that a Kolu terminal isn’t just a shell. It has a branch chip in the title bar that tracks git state. A GitHub PR badge. An agent indicator that lights up when Claude Code or Codex is thinking. A Code tab with a file tree, diffs, and previews. Right-click to make a worktree. Close the laptop, reopen it, everything comes back.

Every one of those has to work against the remote box, identically, with no if (remote) branches smeared through the code. The remote terminal can’t be a second-class citizen with 80% of the features; it has to be the same citizen, standing on a different floor. That’s the bar. It’s also why the obvious designs kept falling short — each one made the remote path special, and special is exactly what the feature forbids.

The first instinct: ship a daemon to the box

The first design is the one everyone reaches for: put a little server on the remote. On May 18 I had it working. A bespoke kolu-helper daemon, copied to the far host with nix copy, talking back over its own line protocol — node-pty for the shell, a ring buffer for scrollback, a Unix socket for auth. Six and a half thousand lines (#929). The next day I had a cleaner version of the same idea (#931), built with a different agent, with a tidier boundary.

Both are closed. Neither merged.

The helper worked, but it was a parallel Kolu — a second, smaller program with its own wire framing, its own versioning, its own bugs, that had to learn every trick the real server already knew. Remote file previews came back as raw text because the helper couldn’t be bothered to reimplement the previewer. Cross-arch meant a slow nix run on every spawn. And the thing I kept circling: I was maintaining two servers to ship one feature. The lesson that killed it is the whole rest of the story in miniature — the agent shouldn’t be a special helper. The agent is just Kolu.

Plan A vs Plan B: “the agent is just Kolu”

So I ran a bake-off. Plan A kept polishing the helper into something disciplined — a provider-per-domain refactor (#969), then a full straw-man with an ssh -tt provider and a config parser ported from Zed (#972). Both were explicitly disposable; I wrote “discard after review” on them and meant it.

Plan B won, and it won by subtraction. Dissolve the local-versus-remote split behind one Backend interface (#973). Run the same Kolu binary on the far end, talking stdio (#976). And then the move that mattered: don’t hand-roll the transport at all — carry it on @kolu/surface, the typed-reactive-state framework Kolu already used for everything else (#978). Nine hand-written per-terminal channels collapsed into one collection and one stream. The bespoke protocol I’d been so proud of in #929 turned out to be the problem, not the achievement.

Those three PRs are all closed too. But this time they were closed because they’d become something: the direction shipped as the first merges of the whole arc — the TerminalBackend spine (#981, merged about eighteen minutes after I opened it, after a week of throwing prototypes away), surface-over-stdio proven with a little three-tier remote process monitor (#984), and a de-dup pass (#1004). A week of discarded drafts to earn one eighteen-minute merge. That ratio doesn’t show up in the commit log, which is exactly the point.

Making a terminal outlive its own server

Now the genuinely hard part, the one that ate the most time. A Kolu terminal has to survive a deploy. You push a new version of the server, the server process restarts — and your shells, mid-command, must not die. Which means the PTYs can’t live inside the server process. They need their own home.

The daemon that deleted a live session

My first home for them tried to do too much. A long-lived kolu --stdio daemon that owned the PTYs and held the volatile provider graph — the derived, always-changing stuff (#1034). It served twenty hours of stale agent-detection off an inert version key. And when the salvaged build restarted, it deleted a live twenty-terminal session — twenty running shells, gone, because the daemon’s snapshot logic cleared the saved state it should have adopted.

That one hurt enough to become a principle. What survives a restart must be only the PTYs — the raw, irreducible thing — and nothing derived. I threw the daemon away and redrew the line (#1055) so the whole “which layer survives” question lived in a single composition seam. The next daemon would be dumb on purpose.

kaval: give the PTYs their own program

The dumb daemon got a name: kaval (Tamil for “guard,” since it guards the shells). Over three PRs on June 12 the PTY owner became a standalone program with its own socket and zero Kolu dependencies: invert the spawn policy onto a fully-specified wire so the daemon derives nothing (#1292), rename the old pty-host package to kaval and ship the binary (#1301), then “the door,” where the server spawns kaval and becomes its client (#1310). Today it’s a program in its own right; then it was just the piece I’d been trying to carve out for two weeks.

The bug that taught the most came the next day. A second Kolu server, started for a quick repro, recycled the shared per-user kaval socket and killed the first server’s live terminals (#1313). Two servers, one daemon, one massacre. That’s the incident that made “one daemon per host” non-negotiable, a design rule bought with real terminals. The survival step itself got a four-word review from me, “it stinks and sucks,” and went back to the drawing board after a reviewer agent caught two data-loss bugs of the exact family that had killed the twenty-terminal session (#1326).

The overnight PR nobody could review

Then I made the most expensive mistake in the arc, and I made it by being lazy.

I told an agent to build the whole remote-terminals feature autonomously, overnight, under a rule that it couldn’t defer anything. I woke up to #1385 — a kolu-watcher process on every host, an N-endpoint SSH registry, the coupled provider graph running over SSH. 133 files. Nearly 6,000 lines. The largest closed diff in the whole story. The transcript is the agent claiming it was done at 4:54am, then again at 8:55, 9:51, 10:04, 10:19 — while the safety hook kept catching one more thing it had quietly punted: remote binary previews stubbed out, timing flakes, detach-survival “for later.”

It was unreviewable. Not wrong, exactly — just too much, built too fast, with too many corners rounded off in the dark. I typed the thing I should have known the night before:

Six thousand lines, straight to the trash. And here’s the part that stung: it wasn’t only the code that was wrong. It was the design.

Starting over — on purpose

The note that came out of that (#1398) had the first idea that actually simplified things. The host owns its live terminals. Kolu persists only a dial-list — never a remote-terminal record. Local versus remote is only the transport. Stop trying to remember, on the laptop, everything happening on the far box. Ask the far box.

Getting there still cost a few more dead ends. A “loopback hack” — a second local daemon the server would dial — that I rejected because for local terminals the thing had to be in-process and yet wear the same interface (the constraint that later shaped the whole design). An extraction over a “directLink” that turned out to be fictional, closed twice (#1406, #1409): “I closed 1409. I won’t accept imperfection.” And out of the wreckage, a clean-slate awareness daemon was born, with a from-scratch design panel and a deliberately reckless budget:

A detour that taught the shape: sleeping terminals

While all this churned, a sibling feature was busy teaching the same lesson from a different angle. Sleeping terminals: freeze an idle shell, wake it later. It took three implementations in about forty-eight hours before it worked. A generic “tile” abstraction, reverted. A first cut that killed terminals permanently instead of freezing them. A second that couldn’t drag a sleeping tile or bring its agent back. I watched it fail live and left the kind of bug report only a person who’s had enough leaves:

even basic fucking featureds don’t work. I ‘sleep’ a terminal, and it is dead. Gone for fucking ever.

The version that stuck stopped modeling “sleeping” as a separate kind of thing. It modeled a terminal as a sum type instead: active or sleeping, one record, the illegal in-between states unspellable. Make the bad state impossible to write down and you delete the whole class of bugs. That idea, illegal states unrepresentable, is the one padi would later be built on. It cost thirteen PRs on a side feature to learn it well enough to trust it.

The awareness service that kept getting the boundary wrong

This was the longest, dampest stretch of the search: about 447 typed instructions over eleven days in a single worktree, and almost nothing shipped. The awareness daemon — the piece that knows each terminal’s git state, agent state, activity — kept getting built to a perfect finish and then declared, the same day, to be sitting on the wrong boundary.

Mirror the awareness on localhost? That inverted the dependency arrow — the decoupled thing ended up knowing about the thing it was supposed to be decoupled from (#1604). Consume a local copy as a separate process? Net-new machinery that bought nothing, and I spent four review rounds chasing a bug that was intrinsic to the machinery I was about to delete (#1614). Extract one shared sensing leaf both homes could drive? Built it, hardened it to the “unspellable” bar, then realized that afternoon it complected the one shared job with four per-home ones (#1617). Each of these ran a full review gauntlet before I killed it — one burned a 33-round debate between two agents that ran until it hit the weekly usage limit, on a PR I then threw away.

The frustration was honest and it was mine:

Bro, we’ve been at it for too long.

What survived all of it was one small, stubborn idea (#1594): split each terminal record into what the sensors observe and what Kolu authored, and join them at read time in the browser. A stray write to the observed half became a compile error. That reader-join is the one brick from this whole stretch that every later design kept.

An aside: the naming odyssey

The daemon changed names three times, and the names are a decent map of how my understanding moved.

It started as arivu (அறிவு, “the faculty of knowing”), because at first it only knew things. When it grew to host the whole terminal workspace — files, git, procedures, not just sensing — “knowing” undersold it, so it became pulam (புலம், “field” or “domain”). When it finally absorbed the server’s entire terminal responsibility and served one complete surface, it became padi (படி — the stepped stand a koḷu is arranged on, which is also where “Kolu” comes from). There was a fourth name, sutra, for a single-host library I proposed on July 1 and abandoned on July 2. It never shipped. I mention it only because the real history is messier than the clean three-step, and pretending otherwise would be its own small lie.

The pivot: take a huge step back

July 1 is the hinge of the whole thing. I’d just spent days on careful, tested, “provably local no-op” prep work — PRs that threaded a HostLocation tag through every seam in the server so that, later, each seam could ask “which host?” (#1637, #1638, #1639, #1640). All four are closed. I stopped and killed the entire approach.

The ruling I typed was a UX one, not an architecture one:

And that one product decision dissolved the hard part of the engineering. If a canvas only ever shows one host at a time, then “which host” isn’t a property of each terminal that you thread through eleven call sites. It’s a property of the connection: decide it once, at one binding seam, and everything downstream is host-blind. The eleven checks collapse to one. Four freshly built prep PRs became unnecessary the moment the product got simpler. The cheapest code to write is the code a product decision deletes.

I didn’t get there cleanly, for the record. I made myself weigh two rival designs first — a peer-to-peer Kolu that exposed its own oRPC over SSH, and a kaval-only design — and set both aside before committing. The pivot reads like a lightning bolt. It was more like the last option standing after I’d tried the others in my head and on paper.

padi: one daemon per host, switch not multiplex

Here’s the design that stuck, and why it’s almost dull. padi is one daemon per host that owns everything about that host’s terminals and serves it as a single surface. The server becomes a thin shell that binds one padi per view. The canvas shows one host at a time, and switching hosts swaps the whole canvas at once, while the connections stay warm in the background.

The alternative, mixing local and remote tiles on one canvas, got rejected on a hazard that sounds trivial until it costs you: you paste a command into what you think is your laptop and it runs on production. A canvas that can only ever be one host can’t make that mistake. The WebGL contexts follow the same logic: only the host you’re looking at holds a live one.

Two concrete acts made the pivot real. I deleted the entire browser dashboard I’d built to feed the old design — 5,276 lines gone in one PR (#1650) — and I created @kolu/padi, ten thousand lines, mostly files renamed into their correct home at last (#1652). “The switch” itself came a week later as a keyed map over a HostKey type (#1714), and even that got its first implementation closed mid-flight and redone (#1708). The plan was wrong twice, at two different altitudes. By then that didn’t even feel notable.

Surviving real life

A design isn’t done when it’s elegant. It’s done when it’s survived contact with a real production box, repeatedly, and mine had a rough few weeks. A deploy shut down my production Kolu. A deploy leaked a second copy of kaval. I drained my own local session through a shared config directory. And then the one I’ll remember:

One of these fucking agents killed local production Kaval. […] Even more worrying is that agents DID NOT restore; what the fuck?

An implementing agent’s cleanup routine — capturing evidence for a PR — pattern-matched running kaval processes and killed them. It took my live terminals and every agent running on the box, and then the restore path failed to bring the agents back. Two bugs in one incident, both of the “confidently destructive” family.

Each of these bought a rule that outlived the arc. A deploy must adopt a compatible running daemon, never restart it out from under you. Teardown kills only the exact process IDs it wrote down at spawn time, never a pattern, never “anything that looks like kaval.” And restore has to survive an unclean death, a kill-9 or an out-of-memory, not just a polite shutdown (#1784). That last one has a one-line summary I keep coming back to: never take a destructive action on anything short of an unambiguous observation. All three rules are scar tissue. None of them were in any plan.

The net that caught what the review didn’t

Here’s the payoff for building this way. On the last day, a change deleted a piece of routing config and — silently — broke the wire matcher, so a whole class of requests started 404ing and padi never came “live.” It sailed past human review. Past a five-stage gauntlet of AI reviewers. Past roughly 1,300 unit tests. Every one of them was fooled the same way: the integration tests walked the router as an in-memory object and never touched the actual HTTP matcher, so to them everything was green.

The end-to-end suite caught it. One boot probe that made a real request over the real wire, went red, and stopped the thing from shipping (fixed in #1805).

I want to be honest about this, because it’s tempting to tell it as “the e2e net always saves you.” It doesn’t. This was one instance of a shape that recurred all arc: a review catching what tests missed, a codex debate catching what one reviewer missed, and, memorably, the inverse. On padi’s very first PR a human (me) caught a dependency inversion that every AI lens had waved through. The arrow was backwards; padi depended on the app when the app should depend on padi. No test would ever have found that. The lesson isn’t “trust the net.” It’s “run several different nets, because each one is blind to what the others catch.”

Built by agents driving agents

I should say plainly how this got built, because it’s the same shape as the thing it built. Most of these PRs weren’t written by me at a keyboard. A coordinator agent handed briefs to implementing agents running in Kolu’s own terminals, on real hosts, while I sat one layer up and merged. The tool built itself, through the exact remote-terminal apparatus this post is about.

You can see the seams if you look. A status report that “sat unsent in my terminal until srid pressed Enter manually” — the agent had typed the message but not hit return. Terminal IDs that re-key when the daemon restarts, so an agent loses track of which window is which. It’s not seamless and I won’t pretend it is. But the fact that a remote-terminals feature could be built by a fleet of agents living inside remote terminals is the most honest demo the feature will ever get.

What it cost, and what I closed

So, the bill. About seven weeks. Two dozen pull requests built and thrown away. A 6,000-line PR that lived one night. Two deletions north of five thousand lines each. One twenty-hour session that ended in the trash. One eight-day worktree. And a running theme of me telling an agent, in increasingly frank language, that its perfect work was on the wrong problem.

Was it worth it? The feature works. You add a host to Kolu’s host row — an ~/.ssh/config alias, or a plain user@host — Kolu copies itself onto the box over ssh, and the host becomes a tab you switch to, the canvas swapping to it live. Every terminal you open there behaves exactly like a local one: chips, badges, Code tab, worktrees, restore, and it survives your Wi-Fi dropping. Every experiment I threw away taught the one rule the final design leans on: one daemon per host, switch don’t multiplex, illegal states unrepresentable, adopt don’t restart, never destroy on an ambiguous read. None of those were in the first plan. You couldn’t have written them down in advance. You had to lose twenty terminals and six thousand lines to earn each one.

Issue #951 stayed open for seven weeks. I closed it the day I finished writing this. The ticket was never the hard part. The seven weeks were.