Kaval Memory — Small Mirror over an On-Disk Transcript

Companion to the kaval heap-OOM RCA. That note bounded the chronic crash; a live look at production (kaval reported at 4740 MB, draining to ~600 MB over minutes) surfaced a second, acute failure it had filed as a red herring. Reproduced on a clean box with the prod versions (@xterm/[email protected]). This is the plan to end both.

Three terms up front, because the rest leans on them:

• mirror — kaval’s own in-memory copy of a terminal’s screen. It’s a headless (display-less) @xterm/headless terminal that replays the exact bytes the real PTY emitted, so the server always knows what’s on screen even when no browser is watching. Today it retains a 10 K-line scrollback (ptyHost.ts:52).
• attach() — the call a client makes to start viewing a terminal (ptyHostSurface.ts). Kaval hands back a one-shot snapshot of the current screen (today, the whole mirror serialized to ANSI, ptyHost.ts:661), then streams live output as deltas.
• reconnect storm — when a client’s WebSocket drops and comes back, it re-attach()es to every open terminal at once, and the in-flight attaches it aborted get reissued: a burst of dozens of attach() calls — and dozens of full-mirror serializes — in one moment.

That single mirror is quietly doing four unrelated jobs: ① a live VT emulator that answers device queries (XTVERSION, DA1/DSR) and scrapes OSC metadata (cwd / title / command); ② the viewport a freshly-attaching client repaints; ③ the deep scrollback kept for PDF export, search, and scroll-back; ④ the wire snapshot, serialized to ANSI on every attach(). Jobs ② and ④ drive the acute spike; job ③ drives the chronic growth — two distinct failures:

	Acute transient	Chronic growth
Cause	attach() serializes the whole mirror (job ④); a reconnect storm fires ~60 at once	the mirror’s deep scrollback (job ③) × an ever-growing terminal count — heap is linear in live-terminal count; terminals are never reaped
Measured	6 concurrent reconnect rounds (60 serializes) → 2.0 GB; 40 rounds → 3.2 GB; drains over minutes (V8 idle reducer)	climbs to the ~4 GB ceiling over days → SIGABRT
Fix	two kaval-only changes defang it (PR1); the snapshot bound finishes it in PR2 — no reload-history loss	the on-disk transcript owns deep history; the mirror shrinks to screen-size

User-facing description

Nothing the user clicks changes. The effects are all in the negative space — what stops happening — plus making deep history honest:

Today	After
Terminals get laggier the longer the server is up; a reconnect can spike kolu by 2–3 GB	flat memory; reconnect costs tens of KB, not gigabytes
kaval OOM-crashes every few days, restarting the server under you	the linear-in-count growth is gone — no scheduled death
Deep scrollback / PDF export / search read the client’s buffer; lost on a cold reconnect	served losslessly from disk, surviving server restarts and updates

Architecture-level changes

The root fix is splitting that one over-loaded mirror so each of the four jobs lands where it belongs:

• Jobs ① + ② + ④ → the hot mirror, made small. Keep a real headless mirror — it stays the VT emulator (job ①, device-query replies + OSC scraping) and the cheap O(1) snapshot source a storm can serialize (jobs ② + ④) — sized to a small line count (rows + the deepest screen-scrape tail any reader asks for). It stays xterm’s native scrollback ring — not a hand-rolled byte-cap: trimming the ring ourselves would reinvent the maintained source-of-truth for a sub-1 MB gain. (Capping by bytes is the transcript’s job — unbounded output volume; capping by a few lines is the mirror’s — screen size.) An idle terminal becomes near-free, without touching the survivability guarantee (terminals freed only on child-exit / user-kill).
• Job ③ → the cold on-disk transcript. Deep scrollback (the reason PDF export, search, restore, and forensics ever needed depth) leaves the heap entirely for a per-PTY append-only store on disk.

The boundary: reuse @kolu/shared/sqlite — don’t hand-roll a storage engine

Durable, seekable persistence is a real volatility — but its receptacle already exists in this repo, so the perfect move is to reuse it, not graduate a parallel one. @kolu/shared/sqlite (shared/sqlite) wraps node:sqlite’s DatabaseSync in WAL mode — the repo’s canonical embedded store, already used by the opencode + codex integrations via withDb + createWalSubscription. And node:sqlite is built into Node 24 (kaval’s runtime), the same “reuse the platform” bet we make for zstd via node:zlib. Extracting a hand-rolled @kolu/segment-log (custom binary format + sparse index + WAL fsync + crc tail-recovery + retention sweeper) would be the parallel hand-rolled store the design philosophy forbids — and a pile of code SQLite gives for free.

What SQLite eliminates: the index (a B-tree on indexed columns), durability + crash recovery (its own WAL journal — a kaval OOM-abort loses at most the last txn), range queries by line / byte / time (an indexed BETWEEN), and retention (a DELETE + incremental vacuum). What stays a kaval leaf (packages/kaval/src/transcript/) is only the terminal-domain glue that imports @xterm/headless and so can’t be agnostic: the DATA/RESIZE/CKPT schema, the byte-offset↔line index, and rendering a range by replaying from the nearest checkpoint. That leaf composes @kolu/shared/sqlite — the electricity split done right: the persistence volatility is already encapsulated; the VT meaning is injected on top.

Implementation details

Two PRs — and only two because the rest is one indivisible change. Shrinking the mirror, writing the transcript, and reading it back are mutually dependent: shrink without read-back regresses copy-all / search / deep-scroll, and read-back without the shrink fixes no memory. So they ship together — there is no point where kolu has less functionality than today.

PR1 Defang the storm — kaval-internal, no disk, no wire change

Shipped #1573. (The PR chip lives here, not in the heading, so the section anchor stays #pr1-defang-the-storm--kaval-internal-no-disk-no-wire-change — an empty JSX node trailing a heading slugs to a stray -.)

User impact: a reconnect storm (rapid reload, network blip, laptop wake — especially with many terminals open) no longer spikes kolu by gigabytes; the worst case drops to a brief, bounded blip. Crucially, a full reload still restores the same history as today — the server still sends the whole mirror snapshot, this PR just stops duplicating and stacking that work. So PR1 is a true strict improvement with no regression — which is why it can ship first, ahead of the disk work. (Bounding the snapshot — the change that finishes the job but would shrink what a reload restores — is deliberately held to PR2, where backfill makes it lossless.)

Two changes, both in attach() (ptyHost.ts:668-688), neither touching snapshot content:

1. Cancellable: after the eager subscribe, if (signal?.aborted) return { snapshot: "", deltas } ("" is wire-legal). A disconnect aborts the in-flight attaches and reissues them; today the aborted half still serializes the full mirror for a reader that’s gone — this makes them no-ops, removing the abort-doubling.
2. Epoch-coalesced snapshot: a per-Entry snapshotCache, cleared synchronously inside the headless.write callback before publish (ptyHost.ts:624-630). Repeated/rapid attaches to one terminal within a single publish-epoch (an idle terminal = a long epoch) serialize once and share the immutable string. Race-free: cache-set and publish-clear are on the same synchronous tap.

Net: the storm drops from the measured multi-GB to roughly (live-terminal count) × one full snapshot — transient tens-to-low-hundreds of MB — with reload history untouched. One honest cost of the memo: it leaves one serialized snapshot pinned per terminal between mutations (an idle terminal’s lingers until its next byte or resize — and getScreenState populates the same slot). That retention is real, but it’s bounded by — and strictly smaller than — the mirror it shadows (a filled 10 K snapshot is ~4 MB of ANSI vs the ~25 MB live cell buffer it serializes), and it’s freed on the next mirror mutation or on teardown — so steady-state heap stays the same O(live-terminal count) class the mirror already occupies, not a new leak. The epoch grain is load-bearing, not a missing release: a reconnect storm’s attaches arrive across many event-loop turns, so only an actual data-parse boundary reliably outlasts the burst — a turn/microtask/timer release would fire mid-storm and re-introduce the N serializes. PR2’s snapshot bound then takes each snapshot — pinned or transient — to ~tens of KB.

Guard: a reconnect-storm test (N concurrent + repeated attach(), assert peak allocation falls to O(terminal-count) full snapshots, not O(storm-size)) — the guard missing today for the exact spike investigated. Channel is untouched (channel.ts:120-131 already correct).

PR2 The on-disk transcript — write, read, and shrink, atomically

User impact: terminals stay snappy no matter how long the server’s been up or how many you have open, and kolu stops OOM-crashing-and-restarting every few days (the chronic death). A full reload now shows the visible screen instantly, then fills in history as you scroll — reaching the full retained on-disk depth (deeper than today’s 10 K mirror, up to the retention cap), rendered at the correct width, and surviving server restarts. No regression window: the snapshot is only bounded — and the mirror only shrunk — once backfill is in place to restore depth, all in this one merge.

Build it bottom-up inside the PR so the bound/shrink is last — the transcript can be exercised by tests before any user path depends on it:

• The transcript/ leaf over SQLite. One DB per PTY ($XDG_STATE_HOME/kaval/transcripts/<id>.db), opened via @kolu/shared/sqlite’s withDb in WAL mode. A single record table — (seq INTEGER PK, kind, firstRow, firstByteSeq, tsMs, cols, payload BLOB) — indexed on firstRow, firstByteSeq, tsMs for the three range queries. payload is a node:zlib zstd-compressed run of coalesced output (batched ~64 KB, not one row per chunk — the write-amplification trap). The writer is the existing proc.onData callback (ptyHost.ts:624-630), so the transcript shares the mirror’s byte stream and inherits attach’s race-freedom — no new race. SQLite owns the index, WAL durability, crash recovery, and range reads; the leaf owns only the schema + the DATA/RESIZE/CKPT framing.
• Checkpoints make backfill correct and cheap (the RCA omits them): without one, rendering any range replays from byte 0 — re-creating the multi-GB spike on every scroll. Every ~K lines write a CKPT row = serialize({ scrollback: 0 }) (viewport + modal preamble, a few KB — not a full-buffer serialize, which would be ~4 MB/checkpoint = gigabytes of index). A range render = restore the nearest checkpoint into a throwaway headless, replay one bounded run, dispose — behind a semaphore so backfill can’t re-storm the read path.
• The read verbs. history({ id, beforeCursor, maxLines, width }) keyed on an opaque byte-offset cursor, not a line number — render-line numbers shift under reflow (reflowCursorLine), so a line-range API bakes in a coordinate that silently moves under resize; a byte offset is reflow-stable, width a render-time parameter. Returns rendered ANSI at the client’s current width; the snapshot frame carries historyCursor (nearest checkpoint ≤ window top) so the join overlaps-not-gaps. Plus exportHistory / searchHistory, extending the existing range-read getScreenText idiom (ptyHostSurface.ts:314).
• The client copy-mode pager. A read-only pager surface that, on scroll past the hot window, fetches older ranges by byte-offset cursor through the stream namespace (rpc.ts, per streaming.md — snapshot-then-deltas, reconnect-safe) and renders them at the pager’s fixed width, re-fetching on resize. PDF export (exportScrollbackAsPdf.ts) repoints at exportHistory; scrollback search at searchHistory. The live Terminal.tsx view is untouched — the pager is a separate surface, so there is no live-grid splice.
• Then, and only then, bound the snapshot and shrink the mirror. With backfill in place, attach() serializes a HOT_WINDOW viewport (serialize({ scrollback: HOT_WINDOW }), the 0.14.0 option; same bound on getScreenState, ptyHost.ts:728) — now lossless, because a reload paints the window instantly and backfills the rest from the transcript on scroll. This is what finally takes the storm to ~tens of KB/snapshot. The mirror itself shrinks to rows + SCRAPE_TAIL_LINES — the screen-scrape promoter’s tail (local.ts:265), a screen-sized constant, not 10 K — so DEFAULT_MIRROR_SCROLLBACK stops being a depth dial. In the same commit, re-route the whole-buffer reads the shrink would otherwise truncate: Copy-terminal-text calls screenText with no range = the whole buffer (useTerminalCrud.ts:280), and scrollback search reads deep — both now read the transcript. The bounded screen-scrape tail (readScreenText(tailLines)) stays on the mirror.
• Retention & privacy as spawn-frame policy (B0-style), not a daemon knob. The spawn input carries a required history: { enabled, retentionBytes }, threaded through the same three paths that carry scrollback today — composeSpawnInput (ptyHost/index.ts:233), kaval-tui’s composeCreateInput, and the in-process host (inProcessPtyHost.ts:260) — so the daemon derives nothing and a missing field is a loud crash. Retention = DELETE oldest rows past the per-terminal byte cap (raise an oldestRow watermark; a sub-floor read returns {kind:"evicted"}, never empty) + a global sweeper that reaps exited-terminal DBs first (live ones never reaped — survivability). enabled:false writes no DB; reads return {kind:"unavailable"}. Files 0600 under 0700 — a perms mismatch is a loud crash.
• Disk fault never kills the PTY. A runtime ENOSPC/EIO degrades that one terminal’s transcript to a surfaced {faulted, lastGoodSeq} (via daemonStatus), never a truncated log shown as complete, never a daemon crash — the one place survivability outranks fail-fast (caught-error-must-not-collapse-to-empty).
• Guards: flat-in-count heap soak (slope ≈ 0); a fast-check lossless round-trip (random write/resize interleavings, replayed per epoch == live grid, concatenated DATA == input); a formatVersion row that fails loud on an unknown schema (distinct from PTY_HOST_CONTRACT_VERSION).

Pinned constants

Constant	Value	Where it lives	Why this value
SCRAPE_TAIL_LINES	40	the mirror-floor formula in ptyHost.ts	Exactly TAIL_REGION_LINES (screen.ts:61), the deepest detector tail any live reader asks for — consumed at agent-adapter.ts:88 and flowing through readScreenText(tailLines) at local.ts:265. Derived from that constant (not a fresh magic number) so the floor can’t drift below a reader’s need — reuse the source of truth.
mirror floor	rows + 40	same	The mirror only owes the screen plus the deepest scrape tail. No OSC handler (7, 0/2, 633;E) or device-query handler reads the scrollback buffer — they extract metadata only — so nothing reads past rows + 40. DEFAULT_MIRROR_SCROLLBACK stops being a depth dial.
HOT_WINDOW	500 lines	the attach() bound and getScreenState, serialize({ scrollback: HOT_WINDOW }) (ptyHost.ts:728)	The bounded attach snapshot. ~500 lines serializes to tens-to-low-hundreds of KB (vs today’s multi-MB full-mirror serialize), repaints the visible screen plus ~10× viewport of over-scroll margin so a reload rarely hits a backfill round in the first gesture, and the pager fetches anything deeper on scroll. Independent of K; tunable.
checkpoint K	one CKPT per ~64 KB zstd DATA block (≈500 lines), deferred to the next clean line boundary	the proc.onData writer (ptyHost.ts:624-630)	Pinned below.

Why K is block-anchored, not a raw line count. The replay-cost spike measured restore checkpoint → resize → replay K lines → render a range on Node 24 / @xterm/[email protected] / [email protected]: K=50: 2.9 ms, K=100: 3.6 ms, K=200: 4.3 ms, K=500: 4.1 ms, K=1000: 6.0 ms. The curve is flat and fixed-cost-dominated — ~2.5–3 ms of throwaway-construct + serialize-restore + reflow, then ~0.003 ms/line marginal. An earlier draft floated a flat K = 1000 lines; the spike refines it to one checkpoint per ~64 KB DATA block because:

• it bounds the real cost driver (bytes parsed, not lines), so a pathological long line can’t blow the replay budget;
• a single huge logical line can never host a checkpoint (the clean-boundary constraint below), so it’s always replayed whole — block-anchoring makes that cheap by letting a render decompress whole blocks;
• it reuses the existing ~64 KB write-batch cadence the transcript already uses, instead of bolting on a second cadence.

Under the DATA-replayed-return rule, worst-case replay distance per page is ≈ K + maxLines + rows ≈ 560 lines → ~4 ms/page render — comfortably sub-frame (< 16 ms) even on a 2–3×-slower production host, and it runs behind the read semaphore so backfill can’t re-storm the read path. Checkpoint storage is negligible: each CKPT = serialize({ scrollback: 0 }) ≈ a few KB, so ~200 checkpoints per 100 K retained lines is sub-MB. K can range 256 (~32 KB) to 1000 (~128 KB) without harm; smaller buys little (fixed cost dominates), larger trims the frame-budget margin.

The backfill seam protocol

Cursor type. byteSeq is an opaque, monotonic byte offset into the per-PTY decoded DATA stream (the running sum of decoded-output byte lengths). It is always minted and consumed at a ground-state clean line boundary — cursor column 0, primary screen, top row is the first row of a logical line (not a wrapped continuation). The record.firstByteSeq index resolves a byteSeq to the ~64 KB block that contains it. Line numbers never go on the wire — they shift under reflow (reflowCursorLine); a byte offset is reflow-stable, width a render-time parameter.

Request (client stream namespace, snapshot-then-deltas):

`history({ id, beforeCursor: byteSeq, maxLines: int, width: int })`

• beforeCursor — the byteSeq at the top of the content the client currently holds. The first backfill uses the snapshot frame’s historyCursor (the byteSeq of the nearest CKPT ≤ hot-window top, already a clean boundary). The server returns the rendered rows immediately above beforeCursor.
• maxLines — a target of physical (rendered) rows, a soft minimum: the server returns whole logical lines totaling ≥ maxLines, so it never splits a wrapped line at the top edge.
• width — the pager’s current render width.

Response: `{ rows: string[] /* rendered ANSI, top→bottom */, nextCursor: byteSeq, atFloor: bool }`

• rows — whole logical lines, ≥ maxLines physical rows (fewer only at the floor).
• nextCursor — the byteSeq at the top of the returned block (== firstByteSeq of the topmost returned logical line; a clean boundary). The client passes it as the next beforeCursor.
• atFloor — true when nextCursor reached the oldest retained byteSeq / eviction watermark. A sub-floor read returns `{kind:"evicted"}`, never empty.

Server algorithm (one call; every render runs behind the read semaphore):

1. Seed checkpoint C = the latest CKPT captured at or before the first line that will be returned. Analytic pick: latest CKPT with capture-line ≤ line(beforeCursor) − maxLines − rows. The −rows margin guarantees C’s own restored viewport (the “seed”) has scrolled entirely above the returned window. Fall back through earlier checkpoints, ultimately the implicit byte-0 checkpoint (a fresh terminal at the initial width).
2. Render the segment: throwaway headless at C.cols → restore C.vtState (serialize restore) → resize(width, rows) (xterm native reflow of the seed) → write the zstd-decompressed DATA run for (C.firstByteSeq, beforeCursor]. Do not replay RESIZE records — this is reflow-to-current. Dispose.
3. Slice: take the last ≥ maxLines rows; snap the top edge up to a logical-line start (the row whose isWrapped === false). nextCursor = that line’s firstByteSeq. Load-bearing invariant: the topmost returned line MUST be DATA-replayed (parsed fresh after C.firstByteSeq), never a reflowed-seed line. If the slice would include a seed line, walk back to an earlier C and re-render.
4. Return rows + nextCursor + atFloor.

Client backward-paging + stitch.

• On attach, the snapshot carries historyCursor (nearest CKPT ≤ hot-window top). Render the HOT_WINDOW snapshot; its top == historyCursor.
• Scroll up past the loaded top: keep a cursor stack; call `history({ beforeCursor: topCursor, maxLines: viewportRows × overscan, width })`; prepend rows; push nextCursor; set topCursor = nextCursor; repeat; stop at atFloor.
• Resize the pager: discard rendered rows, keep the cursor stack (byte offsets); re-issue history() from the bottom cursor at the new width and rebuild upward. Because every cursor is a clean ground-state boundary, the same byteSeq re-renders to a clean physical-row boundary at any width — no row duplicated or dropped across the resize.

No-gap / no-overlap. Page k covers exactly [nextCursor_k, beforeCursor_k) in byteSeq, snapped to clean line boundaries, and beforeCursor_k == nextCursor_{k−1} (adjacent pages share the boundary). A ground-state boundary maps to a physical-row boundary at every width, so concatenation reconstructs the single-shot render with no duplicated or missing row. The hot↔cold join: the first history() uses beforeCursor = historyCursor (the snapshot top, a CKPT byteSeq), so its rows end exactly at the snapshot top — prepend with no overlap. exportHistory routes the same machinery in FAITHFUL mode (restore@C.cols, replay DATA + RESIZE, render at the historical width, per resize-epoch); searchHistory ranges over the same index.

Cross-width reflow verdict — YES, byte-identical, under two constraints. A checkpoint-rooted segment render is byte-identical to the global single-shot reflow at the client’s width W, validated across W ∈ {40, 80, 100, 120, 160}, every range, a 4000-char wrapped logical line, and wide-CJK / emoji / combining-mark content — provided:

• Constraint 1 — checkpoint placement. A CKPT is captured only at a ground-state clean line boundary: buffer.getLine(baseY).isWrapped === false (viewport top is the first row of a logical line), cursor at column 0, primary screen. The writer marks a checkpoint “due” every K worth of bytes but defers capture to the next byte boundary where isWrapped(baseY) === false. Why: serialize({ scrollback: 0 }) preserves logical lines (it joins wrapped rows), so a terminal re-wraps them at its own width — faithful only if each captured logical line is complete. The viewport bottom is always completed by DATA replayed after the checkpoint; only the top can be irreparably partial, so forbidding a wrapped top closes the only hole. The same constraint governs nextCursor, guaranteed by snapping to a logical-line start.
• Constraint 2 — return from DATA. Every returned line must be DATA-replayed; the checkpoint is only a VT seed for content above the window, and its restored-viewport lines are never returned. This neutralizes the one imperfection found (below).

The copy-mode pager

A read-only, fixed-width reader surface — never a splice into the live reflowing grid. The live Terminal.tsx is untouched; it stays mounted and running behind the pager the whole time.

Invocation — three entry points, one state owner.

• Keybinding viewHistory = Mod+Shift+H (Cmd+Shift+H mac / Ctrl+Shift+H Linux/Win), added to _ACTIONS in actions.ts. Verified clean: KeyH is unused across the whole ACTIONS map, and Mod+Shift+<letter> is the established app-chord convention that clears in-PTY bytes — same family as shuffleTheme (actions.ts:279), toggleDock (actions.ts:311), toggleCanvasPosture (actions.ts:320). It is not in PROHIBITED_KEYBINDS (only Ctrl+B and Ctrl+J are reserved, prohibitedKeybinds.ts:21-33); keyboard.test.ts proves no collision. Set focusScopeMarker: TERMINAL_SEARCH_MARKER (actions.ts:158) exactly like findInTerminal (actions.ts:215-227) so the chord is claimed only inside a terminal subtree — Firefox’s own Ctrl+Shift+H still works everywhere else. Add toggleHistoryPager: (id) => void to ActionContext.
• Command palette — “View terminal history” in the active-terminal section of commands.tsx:234-369, beside “Export scrollback as PDF” (commands.tsx:284-289), via actionPaletteCommand("viewHistory", …) (actions.ts:399-413). Offered on both the active and sleeping arms — history is disk-backed ($XDG_STATE_HOME/kaval/transcripts/<id>.db), so it needs no live PTY — but gated on a new meta.history?.enabled flag so an opted-out terminal never offers a dead pager.
• Title-bar button — a clock / History icon in TileTitleActions.tsx between Find (140-152) and Screenshot (153-165), reusing TILE_BUTTON_CLASS, Tip, and the onTile select-then-act wrapper (63-67). Add a HistoryIcon to Icons.tsx (a clock with a counter-clockwise arrow). Register a one-time tip in settings/tips.ts per the Feature-Discoverability rule.

State owner: a useHistoryPager singleton built with createSharedRoot, a near-clone of useTerminalSearch.ts:31-78 — per-TerminalId open-state, openFor(id) / isOpen(id) / close(), and an on(store.activeId) effect that closes the pager on active-terminal switch (the find-bar contract).

Close: Esc / backdrop (desktop) or drag-down / backdrop tap (mobile), the ✕, or “Jump to live ↓”. Every close routes through withKeyboardDismiss and returns focus to the live grid via refocusTerminal (ModalDialog.tsx:25-39, opted in with refocusOnClose).

Desktop mockup — a tall ModalDialog size="lg" reader, backdrop dims the whole canvas so it reads as a separate surface:

┌──────────────────────────────────────────────────────────────────────────────┐
│  ⏱  History — claude-code (feat/female-flat)      [Aa] [.*]  Find… 3/57  ‹  › │
│                                          Jump ▾    ⤓ PDF              ✕  Esc    │
├──────────────────────────────────────────────────────────────────────────────┤
│ ····· Older output trimmed to stay under the 256 MB history limit ··········· │  ← evicted sentinel
│  $ pnpm test                                                                   │
│  ✓ packages/kaval   (42)                                                       │
│  ✓ packages/shared  (18)                                                       │     read-only xterm
│  …                                                                             │     (DOM renderer, no WebGL)
│  $ git log --oneline -5                                                        │     fixed width = pager cols
│  3eb6cc8  fix(kaval): defang the reconnect storm (#1573)                       │     text is selectable → copy
│  ▓▓▓ match: "reconnect storm" highlighted in view ▓▓▓                          │
│  …                                                                             │
│                                                                                │
├──────────────────────────────────────────────────────────────────────────────┤
│  line ~1,284,330 · 3 days ago · 132 cols                ↓ Jump to live   ◴ new │  ← footer; ◴ ring pulses
└──────────────────────────────────────────────────────────────────────────────┘

  Top-sentinel variants (whichever applies as you scroll up):
    backfilling →   ····· ⠋ loading older history… ·····
    true start  →   ──────────── Beginning of session ────────────
    disabled    →   History isn't being recorded for this terminal.
    disk fault  →   ⚠ History may be incomplete — disk error; showing up to the last good point.

  Jump ▾ menu (anchored popover):     Search field toggles:
    ┌──────────────────────┐           [Aa] case-sensitive   [.*] regex
    │ ⤒  Top of history    │           (chips inside the field, ripgrep-style)
    │ ⤓  Latest            │
    │ ─────────────────    │
    │ 🕘 1 hour ago        │           NOTE: no "jump to line N" — render-line
    │ 🕘 Today 09:00       │           numbers shift under reflow, so the only
    │ 🕘 Yesterday         │           position anchors are TIME and top/latest
    │ 🕘 Pick a time…      │           (the byte-offset cursor is opaque).
    └──────────────────────┘

Mobile mockup — a @corvu/drawer side="bottom" sheet at h-[90vh] (the RightPanelDrawer pattern), grabber pill, ≥44 px tap targets:

              ▁▁▁▁▁▁                                   ← grabber (w-10 h-1 rounded-full)
   ┌────────────────────────────────────────────┐
   │ ⏱ History                 Find…    ⤓    ✕   │     header — each control ≥44px
   │ claude-code · feat/female-flat              │
   │ [ Top ]  [ Latest ]  [ Jump to time ▾ ]     │     jump row (full-width buttons)
   ├────────────────────────────────────────────┤
   │ $ pnpm test                                 │
   │ ✓ packages/kaval (42)                       │     read-only xterm
   │ …                                           │     native touch scroll;
   │ ▓ match highlighted ▓                       │     backfills on scroll-near-top
   │ …                                           │
   │                                             │
   ├────────────────────────────────────────────┤
   │  line ~1,284,330 · 3 days ago               │
   │        ↓   Jump to live   ◴                 │     big target, ring pulses on new output
   └────────────────────────────────────────────┘
        (Drawer.Overlay backdrop dims the tile behind)

The component forks on layoutMode() (useMobile.ts:58-67): ModalDialog for desktop, Drawer for phone/compact — exactly as the right panel forks between a Resizable split and RightPanelDrawer. Touch invocation is a “History” row in MobileChromeSheet.tsx:87-141 next to Palette/Settings/Inspector (with a Kbd chip via formatKeybind), plus the palette. Every control is ≥44 px (above the 24 px WCAG floor the codebase already targets, MobileTileView.tsx:191-211).

Interactions.

• Scroll / page / backfill. The body is a dedicated read-only @xterm/headless-on-DOM instance (its own FitAddon + SearchAddon, DOM renderer to spare the WebGL budget, webglBudget.ts), never attached to the live PTY. It opens at the bottom, contiguous with the screen the user just left (the snapshot’s historyCursor makes the join overlap, not gap). Reading is upward. Bindings: wheel/trackpad, PageUp/PageDown, ↑/↓, plus pager idioms — Space/b page down/up, g/G top/latest, / focuses search. Within ~N rows of the loaded top it fires `history({ id, beforeCursor: topCursor, maxLines, width })` and prepends the returned ANSI, holding scroll position stable by re-anchoring on the previous top line’s cursor (byte cursors are reflow-stable; line numbers are not). A thin top sentinel shimmers in flight.
• Search within history (two-tier). The header field mirrors SearchBar.tsx:35-184 — “n / m” count, up/down chevrons, Enter = next, Shift+Enter = previous, Esc closes. But finding goes through searchHistory({ id, query, regex, caseSensitive }) (cross-window seeking on disk), not xterm’s in-buffer SearchAddon. Selecting a match fetches and centers the range; once in view, the pager’s own SearchAddon paints the in-view occurrences with the live search decorations (SEARCH_OPTIONS, SearchBar.tsx:22-32). Clean split: searchHistory seeks, the local SearchAddon highlights.
• Jump ▾ — an anchored popover (surface() chrome) offering Top (g/Home), Latest (G/End), and time anchors (relative quick-picks + “Pick a time…”) because the transcript is time-indexed (by-time range reads at 0.004 ms). No jump-to-line — render-line numbers drift under reflow, the whole reason the cursor is an opaque byte offset.
• Export PDF — the “⤓ PDF” button repoints exportScrollbackAsPdf.ts:19-89 from the in-buffer serializeAsHTML() (today clipped to the 50 K client ring, config.ts:29) to exportHistory({ id, width, fromCursor?, toCursor? }) — full on-disk depth, rendered per the rules in Full PDF export below. A live text selection offers “Export selection”.
• Copy — native selection over the read-only xterm; Ctrl+Shift+C / Cmd+C reuses copySelection (actions.ts:294-302). An overflow “Copy all history” calls the same transcript text read that handleCopyTerminalText is re-routed onto (useTerminalCrud.ts:275-293) — palette “Copy terminal text” and the pager share one disk-backed source.
• Coexistence with the live tile. New PTY output keeps accruing to the mirror and the transcript while you read; the pager doesn’t tail it (it is the past), but a “◴ new” ring on “Jump to live ↓” pulses when output arrived, reusing ScrollToBottom’s animate-ping ring (ScrollToBottom.tsx:30-33). “Jump to live ↓” closes the pager and scrolls the live grid to the bottom.
• State edges. Reconnect: backfill rides the reconnect-safe stream namespace; onRetry clears the pager xterm (terminal?.reset()) before the re-subscribed snapshot lands — the same double-paint defense the live attach uses (Terminal.tsx:795-808). Evicted: “Older output trimmed to stay under the NN MB history limit” — the honest `{kind:"evicted"}` state, never silent-empty. Unavailable (history.enabled:false): “History isn’t being recorded for this terminal” (`{kind:"unavailable"}`). Faulted: on a runtime disk fault the daemon degrades that one transcript to `{faulted, lastGoodSeq}` via daemonStatus; the pager shows a top warning banner, never presenting a truncated log as complete — the one place survivability outranks fail-fast (caught-error-must-not-collapse-to-empty).

Reused primitives: ModalDialog (desktop host, size="lg", height min(74vh, 60rem)); RightPanelDrawer.tsx:58-85 (mobile host — grabber, overlay, withKeyboardDismiss + restoreFocus={false}); SearchBar.tsx:35-184 (header field clone); ScrollToBottom.tsx:7-37 (jump-to-live ring); Terminal.tsx:34,795-808 (streamCall(client.stream.history, …, { signal, onRetry })); useTerminalSearch.ts:31-78 (useHistoryPager singleton); webglBudget.ts (DOM renderer); Surface.ts + Kbd.tsx (popover chrome + chips).

Full PDF export (product question — answered)

Will users get the full PDF instead of a clipped one? YES, decisively. Today exportScrollbackAsPdf.ts:28 serializes serializeAsHTML() off the live client xterm ring — clipped to DEFAULT_SCROLLBACK (50 K) and in practice only to what that client buffered (the header even admits it is “NOT the full session”). After PR2, useTerminalCrud.exportScrollbackPdf (useTerminalCrud.ts:325) repoints at exportHistory, which reads the entire per-id transcript on disk — bounded only by the per-terminal retentionBytes, surviving server restarts. A multi-hour session that scrolled past the client cap exports in full.

Render width = historical-per-span (faithful), not a fixed export width and not the live width. exportHistory walks the transcript oldest→newest and renders each inter-RESIZE span at the cols actually in effect then, freezing each span’s wrapped lines before the next RESIZE is applied. A PDF is an archival document; a 200-col table re-wrapped to a narrow width is the exact content the user exported to keep, turned to garbage. The RESIZE epochs make per-span width recoverable (each CKPT stores cols), and the spike proved checkpoint-replay reproduces the exact screen across resize with RESIZE records load-bearing (the negative control) — using them is reuse the source of truth. When the pager is the export source, “current width” = pager width for the visible-range case; the full-depth archival export uses historical-per-span. (This is a desirable divergence from a live xterm, which reflows all scrollback to one final width.)

Theming handoff (preserving exportScrollbackAsPdf.ts’s existing “server headless has no theme” constraint): the server streams per-span rendered ANSI-with-SGR segments through the stream namespace (a finite ordered stream, idempotent restart on reconnect, so deep depth isn’t one giant message); the client writes each segment into an offscreen @xterm/headless + SerializeAddon carrying the live theme — resizing it to each segment’s historical cols before writing — then serializeAsHTML() → the same themed print window the file opens today. Depth moves to the server (un-clipped, faithful); theme stays client-side.

Edge — history.enabled:false. No transcript, so deep export falls to the live client buffer exactly as today (clipped). Not a fail-fast violation: nothing required is missing — the user chose no retention, and exporting the visible buffer is the correct behavior for that mode. The default is enabled:true, so full-depth is the path almost every terminal takes.

Mobile resize (product question — answered)

A mobile (or any narrow) client resizing a terminal is harmless to scrollback and PDF, because width is render-time for reads and historical-per-span for export.

Width is one shared PTY size — pre-existing, unchanged by PR2. The resize path: client Terminal.tsx FitAddon → client.terminal.resize({id,cols,rows}) → router.ts:113 → PtyHostTerminalProxy.resize (local.ts:161) → ptyHost.resize (ptyHost.ts:754) → entry.proc.resize (the one node-pty child) + entry.headless.resize (the one shared mirror) + invalidateSnapshot. There is exactly one proc/headless per terminal — no per-client size. All attached clients share the width, last-write-wins; a mobile client fitting to 40 cols reflows the live grid for everyone via SIGWINCH. This is today’s behavior, not introduced by PR2.

What PR2 adds and why it stays correct:

• RESIZE journaling: ptyHost.resize gains, beside the proc/headless resize + invalidateSnapshot, an append of a typed RESIZE record (kind=RESIZE, cols, rows) at the current byteSeq — recorded at its true stream position, the load-bearing record the negative control validated. It lands on the same proc.onData/resize writer path, inheriting attach’s race-freedom.
• history() reads render at the reading client’s current width (a render-time parameter), keyed on the reflow-stable byte cursor. A desktop client backfills the same byte range at 200 cols while a phone backfills it at 40 — both get full depth, neither clips the other, and because the cursor is a byte offset, a resize never moves a client’s scroll position.
• Live snapshot on attach (serialize({ scrollback: HOT_WINDOW })) reflects the current shared width; a desktop attaching while a phone holds the PTY narrow paints narrow for a frame, then its own FitAddon re-fits and resizes the shared PTY back (existing behavior). Transcript / history / PDF untouched.
• PDF is historical-per-span, so a mobile-narrow span renders at 40 and a desktop-wide span at 200 in the same export — the document is never clipped or squeezed to a narrow live width. A mobile resize cannot shrink the exported document.
• The pager is its own fixed-width surface. A phone rotating or a foldable unfolding just recomputes the pager xterm’s cols (its own FitAddon) and re-fetches the visible range at the new width — it never reflows or corrupts the live grid and never writes anything back to the transcript (typed records are width-agnostic; width is render-time only).

searchHistory semantics

A server-side scan over the on-disk transcript that re-routes the find-bar’s deep search — not the live xterm SearchAddon.

• Default = literal substring, case-INsensitive — exactly what SearchBar.tsx’s SEARCH_OPTIONS does today (it sets only incremental + decorations; regex/caseSensitive unset → false), so the re-route changes nothing the user sees.
• regex and caseSensitive are opt-in booleans on the input, mirroring xterm’s ISearchOptions shape 1:1 (reuse the source of truth — these are genuine search capabilities, not degradation knobs, so the fail-fast “no override knobs” rule does not apply). The find-bar can expose toggles later; until then the default reproduces today exactly.
• Invalid user regex is surfaced as a find-bar validation error (“invalid pattern”), never collapsed to empty results (a malformed pattern is user input, a surfaced error, not an internal swallow).
• How it scans: the schema keeps only the zstd VT payload (no plain-text column — that is write-amplification and a second source of truth), so search replays rather than greps. It walks record rows newest-first from beforeCursor (or the tip); for each zstd DATA block it decompresses and replays from the nearest preceding CKPT into a throwaway headless — the same machinery history()/exportHistory use — joins isWrapped continuation rows into logical lines (so matching is width-independent: a hit is never split by an arbitrary wrap column), and applies the literal-substring or RegExp test per logical line. Runs behind the same read semaphore as history().
• Returns an ordered (newest-first) bounded array: `{ cursor: ByteSeq (reflow-stable, feeds history() so the pager opens AT the match), firstRow: Row, text: string, matches: [{ start, end }] }` + nextCursor: ByteSeq | null + truncated: boolean. Capped at maxResults (hard cap 1000) per call; on hitting the cap it sets truncated and returns nextCursor so the find-bar pages “search older”. One-shot request/response RPC, cursor-paged — only the paging history() backfill needs the stream namespace; a search reconnect just re-issues the query.

Session restore semantics

The visible-screen restore is unchanged; the transcript only adds deep, durable reach — it does not resurrect a killed PTY’s live screen.

• Warm restart (daemon outlives kolu-server — the common case, reconcile.ts → adoptTerminal): the live PTY and its in-kaval mirror survive in the daemon’s memory, so the visible screen repaints from the surviving mirror via re-attach exactly as today. PR2 only shrinks that mirror to rows + 40 and bounds the snapshot to HOT_WINDOW — and the reload now reaches more history (full on-disk depth via backfill), not less. No regression.
• Cold restore (no survivor — the “restore card” re-spawns onto a fresh daemon): the re-spawn reuses the saved terminal id (id stability is already the architecture’s invariant — kolu-server mints terminal id == PTY id), so it reopens the same <id>.db and appends (seq continues from the persisted max). The fresh shell is blank on screen (only the existing agent-resume replay repopulates it), but the pager/search/PDF reach the pre-restart deep history through the continued transcript.
• Sleep→wake: wake re-spawns the PTY fresh on the same id and reopens/appends the same <id>.db, so a woken terminal’s pager scrolls back into its pre-sleep history — reach sleep/wake lacked. Live screen = the fresh resumed shell (unchanged).
• The one-time 4.0 recycle: the breaking PTY_HOST_CONTRACT_VERSION 3.3→4.0 bump (ptyHostSurface.ts:74) kills every live terminal once. Their transcripts persist on disk but the terminals are gone — exited-terminal DBs the global sweeper reclaims first. Restore does not resurrect them; durable history begins for terminals spawned at/after the recycle.

node:sqlite inside the real nix closure — confirmed

It loads, two ways.

1. Analytic closure chain. nix/nixpkgs.nix pins nixpkgs rev f8573b9c935cfaa162dd62cc9e75ae2db86f85df; nix/overlay.nix adds only kolu-fonts and does not override nodejs; default.nix’s kaval derivation wraps ${pkgs.nodejs}/bin/node with the tsx loader and bin.ts (no extra flags, no NODE_OPTIONS). So the kaval runtime is exactly pkgs.nodejs, which at that pin resolves (nix eval locally + a clean nix build on a box) to /nix/store/sy0c7j0npsq33d9zhnnzvjnzc52f4y0p-nodejs-24.13.0. node:sqlite is a C++ builtin compiled into that binary, so the full-closure build cannot change the result.
2. Empirical, on a fresh pu box (destroyed after; egress probed 200): built that exact pinned pkgs.nodejs + pkgs.tsx, then ran node:sqlite three ways under …-nodejs-24.13.0/bin/node, all exit 0 — (a) bare require("node:sqlite") → DatabaseSync is a function; (b) a real WAL DB: PRAGMA journal_mode=WAL → {journal_mode:wal}, 1000 zstd-magic BLOB rows inserted, indexed range-by-firstByteSeq SELECT returned the right row, BLOB round-tripped as Uint8Array; (c) kaval’s exact launch shape node --import <tsx loader.mjs> file.ts importing node:sqlite from TypeScript — works, the tsx loader passes the builtin specifier through.

No flag, no NODE_OPTIONS, no suppression knob. Node 24.13.0 exposes node:sqlite by default (the --experimental-sqlite requirement was dropped in v23.4); both require() and ESM import return exit 0 with zero flags, including through tsx. The only runtime effect is one ExperimentalWarning: SQLite is an experimental feature… printed to stderr on the first DatabaseSync construction. The implementer must leave the kaval wrapper’s --add-flags unchanged and must NOT add --no-warnings / NODE_NO_WARNINGS / --experimental-sqlite: a suppression knob is exactly the override the fail-fast rule forbids, it would mask unrelated warnings, and the existing codex/opencode integrations already import { DatabaseSync } from "node:sqlite" and emit this same warning in the shipped server with no special handling — PR2 follows that precedent. Net: import { DatabaseSync } from "node:sqlite" in @kolu/shared/sqlite, do nothing to the nix wrapper, accept the one cosmetic stderr line.

Watch-item (not an open question): node:sqlite is officially experimental — the API “might change at any time.” The surface used (new DatabaseSync(path), prepare/run/get/exec, PRAGMA journal_mode=WAL, BLOB↔Uint8Array) is stable across the 24.x line and the version is locked by nixpkgs via npins, so any Node bump rides a deliberate npins update, never silent drift. A future major could change the API — the implementer’s to watch, not a blocker.

---

Open questions: NONE. Every prior fork is closed and every constant pinned. The single artifact surfaced by the seam spike is not an open decision: the combining-mark-on-width-change misattachment is an upstream xterm reflow limitation, reproduced by a plain live-terminal resize with no checkpoint involved (so the pager is not a regression), and the protocol’s DATA-replayed-return rule keeps it out of every returned and stitched line — it can only touch a reflowed checkpoint seed, which is never displayed. Stated here in the open rather than papered over; it gates nothing in PR2 and a future xterm fix removes even the seed-level trace.