Status: implemented, independently reviewed, and mandatory.
spawn-worker plus spawnd supervision is the only production session path.
There is no backend selector, per-session escape hatch, or fallback. The accepted
cutover boundary and old-session drain procedure are in
TMUX_REMOVAL.md. Where this document and code drift,
daemon/src/sessiond/ and daemon/src/worker_backend.rs are authoritative.
Governing trust document: TRUST.md. Every design choice below is tied back to it; the short version is that tmux was the last piece of infrastructure between the session process and the browser that (a) we don't control, (b) holds terminal plaintext at rest, and (c) contains a full server-side terminal emulator we neither need nor test.
1. Why replace tmux
tmux earned its place: detached sessions gave us session survival across
spawnd restarts for free. But it fights the operator model and the
product in specific, structural ways:
| Problem | Consequence |
|---|---|
| tmux server holds scrollback in plaintext, in memory and with no at-rest encryption story | the one place terminal content persists on the host is the one place we can't encrypt |
| tmux is a terminal emulator in the middle: session → tmux grid → re-emitted escape codes → browser xterm.js | double emulation causes fidelity bugs (repaint gaps, mouse/copy-mode weirdness) that we've been patching with refresh-client and SIGWINCH hacks |
| one tmux server per user socket owns all sessions | a tmux server crash or wedge kills every session on the host at once |
control is subprocess-based (tmux send-keys, capture-pane, pane_in_mode polls) | per-keystroke subprocess costs we had to cache around (copy_mode_cached), and content transits argv/pipes of a third-party binary |
| copy-mode intercepts input | browsers had to detect-and-cancel it (tmux::cancel_copy_mode) because a remote viewer must never be trapped in a server-side mode |
| scrollback is line/grid-based | replay re-serializes the grid instead of replaying the byte stream the browser's emulator actually consumed |
The replacement keeps the two properties tmux actually gave us — session survival across spawnd restarts and reattach with history — and drops everything else.
2. Design principles
- The session terminal path is endpoint-owned. The worker protocol stays on a
protected local endpoint — a Unix socket in a
0700directory or an owner-only Windows named pipe — and the browser's low-latency copy travels over mandatory WebRTC DataChannels.spawndsends only content-free activity and signaling/lifecycle JSON over its server control socket. - User-facing terminal rendering lives in the browser. xterm.js in
web/owns the grid a human sees. The worker also holds a headless emulator (sessiond/emulator.rs, alacritty'sTermcore plus an owned ANSI serializer) fed from the PTY read path. Its current primary and alternate-screen grids are plaintext state resident for the worker's lifetime, bounded by the active terminal geometry plus a bounded history drain window. It synthesizes the live-screen repaint served with every replay and decides which lines commit to history; it never transforms the live forwarded bytes. This is a deliberate revision of the original "workers are byte pipes" rule: the byte-pipe design needed a SIGWINCH jiggle to provoke checkpoint repaints from the app, which disturbed the session, stacked duplicate frames into scrollback on every rotation, and made checkpoint quality depend on each app's WINCH behavior. The emulator's fidelity is a tested contract (feed → serialize → re-feed ⇒ identical state), not an assumption. - Raw PTY bytes in the live path; committed lines in the history path. Live output is forwarded byte-for-byte, unparsed. History is NOT the byte stream: raw TUI bytes are a rendering protocol, not a document, and re-executing them can never yield faithful scrollback (intermediate repaint frames, resize reflow). Instead the emulator commits each line exactly once — at the moment it scrolls off the screen — serialized as styled text, and only those committed lines are logged. Replay is "committed history + a freshly synthesized screen repaint."
- One process per session. Crash isolation, per-session keys, per-session lifecycle, no shared mux server.
- Honest crypto claims. Encrypted-at-rest scrollback protects the segment files. It does not mean "encrypted before DRAM": the PTY path, emulator grid, committed-line serialization, replay, and forwarding all require plaintext in host memory (§6.3).
3. Process model
spawnd (host supervisor, one per host)
├── ws/rtc: content-free signaling/lifecycle; WebRTC peer connections
├── worker_backend: launch / adopt / signal workers
│
├── spawn-worker --session-id A … (one process per session, independent tree)
│ ├── owns the PTY master (portable-pty)
│ ├── session process — the user's login shell (session leader on the PTY slave)
│ ├── headless emulator (grid state + bounded history drain window)
│ ├── encrypted scrollback log (ChaCha20-Poly1305, segmented)
│ ├── lifetime reservation: flocked fd (Unix) / exclusive HANDLE (Windows)
│ ├── supervisor endpoint: Unix stream / byte-mode Windows named pipe
│ └── lifecycle endpoint: Unix datagram / message-mode Windows named pipe
└── spawn-worker --session-id B …
- spawnd stays the host supervisor. It owns the control-plane connection, WebRTC, and session registry. It launches workers, adopts orphaned ones, routes input/output, and reaps exits.
spawn-worker(daemon/src/bin/spawn-worker.rs, logic indaemon/src/sessiond/worker.rs) is one process per session. It binds its socket, waits forStart, spawns the session's login shell on a PTY it owns with portable-pty, and from then on: streams output, accepts input/resize, maintains the scrollback log, serves replay.- On Unix the worker is spawned with
process_group(0). Under systemd, spawnd's unit needsKillMode=processor workers are killed with the cgroup on restart. On Windows it is launched withCREATE_BREAKAWAY_FROM_JOBandDETACHED_PROCESS; detaching prevents a classic console host from flashing before the worker creates the session's headless ConPTY, and failure to break away is fatal rather than silently weakening session survival. It then puts itself and subsequently created ConPTY descendants in an unnamed kill-on-close Job Object. On both platforms its fate is tied to the session PTY, not to spawnd, so supervisor restarts leave the session running. - Worker runtime cost is scoped per session: a tokio runtime pinned to 2 threads, two blocking PTY I/O threads, and headless primary/alternate screen grids whose size follows the current terminal geometry. It does not retain a session-length plaintext grid history.
Unix filesystem layout
worker_dir() resolves $SPAWND_WORKER_DIR → $XDG_RUNTIME_DIR/spawn/workers
→ <config_dir>/workers, created 0700:
workers/
<session-id>.lock # lifetime exclusive reservation (0600)
<session-id>.sock # ordinary supervisor listener
<session-id>.lifecycle.sock # atomic fixed-size TERM/KILL datagrams
<session-id>.scrollback/ # 0700; seg-00000001.log … (ciphertext only, 0600)
The directory is ownership-checked and forced to 0700; failure is fatal.
Both sockets and the reservation are ownership-checked and forced to 0600.
spawnd acquires the per-session flock before it spawns a worker and exposes the
locked fd only in that post-fork child. The worker holds it for its lifetime.
Duplicate creates therefore fail before a second worker is spawned. A lock
released by a crash authorizes stale-socket recovery; no code unlinks an
endpoint while the lock says a worker may still own it. Each endpoint cleanup
also compares the socket's recorded device/inode identity, so a delayed old
cleanup cannot unlink a replacement at the same pathname.
Preferring XDG_RUNTIME_DIR puts sockets and ciphertext on tmpfs where
available: gone on reboot, never on spinning rust. That is a feature — the
scrollback key is process-ephemeral anyway (§7).
Windows endpoint and metadata layout
Windows uses $SPAWND_WORKER_DIR as an exact combined test/operator override.
Without that override, endpoints and logs have separate non-roaming roots;
the worker receives the metadata directory explicitly and never infers it from
the log directory. Protected current-user-only DACLs are applied throughout:
%LOCALAPPDATA%\spawn\state\<instance>\workers\
<session-id>.lock # exclusive-open lifetime reservation
<session-id>.endpoint # protected discovery marker, not proof of liveness
%LOCALAPPDATA%\spawn\logs\<instance>\workers\
<session-id>.scrollback\ # ciphertext segments plus detached worker.log
<instance> is the stable eight-hex tag derived from the canonical config
root. Endpoint discovery scans only the state tree; encrypted scrollback and
detached stderr stay in the log tree. With $SPAWND_WORKER_DIR, both sets of
files retain the earlier single-directory layout beneath the supplied path.
The supervisor opens .lock with share mode zero and transfers that exact
HANDLE through PROC_THREAD_ATTRIBUTE_HANDLE_LIST; the worker validates its
final path, file ID, type, owner, and protected DACL, clears inheritance, and
retains it for life. Only that reservation plus three NUL standard handles are
inherited. A crash leaves .endpoint but releases the reservation; a new owner
may remove the stale marker only after it acquires the reservation. Cleanup is
guarded by volume serial plus 128-bit FILE_ID_INFO, so an old worker cannot
delete a replacement marker.
Each worker owns two flat local names because Windows pipe-name components cannot contain backslashes:
\\.\pipe\spawn-<current-user-SID>[-<8hex-config-tag>]-<session-uuid>
\\.\pipe\spawn-<current-user-SID>[-<8hex-config-tag>]-<session-uuid>-lc
Both pipes reject remote clients, use first_pipe_instance for their bootstrap
instance, carry a protected DACL whose only allow ACE is the current user, and
validate the peer process token's user SID before protocol I/O. The main pipe
is byte mode and retains the framing below. Its listener rolls a replacement
instance before serving the connected candidate. The lifecycle pipe is message
mode with eight instances: one listener and at most seven fixed-size handlers;
silent clients expire after 100 ms.
Windows PTYs use portable-pty's ConPTY backend for I/O and resize. START
resolution uses the complete supplied PATH and PATHEXT; resolved .cmd and
.bat programs are wrapped through %ComSpec% (cmd.exe fallback) with
command extensions enabled, delayed expansion disabled, and batch arguments
quoted against cmd metacharacters. Native .exe/.com programs launch
directly. ConPTY's UTF-8/VT output is fed unchanged through the existing live,
emulator, and encrypted-history path—host-generated clear, cursor, title, wrap,
and mode sequences are not stripped or normalized.
4. spawnd ↔ worker wire protocol
daemon/src/sessiond/wire.rs. Length-prefixed frames on the Unix stream or
Windows byte-mode named pipe:
+-----------+------+-----------------+
| len (u32) | type | payload |
| LE | u8 | len bytes |
+-----------+------+-----------------+
len counts the payload only; MAX_FRAME_LEN = 32 MiB (replay dominates and
is capped far below this by the scrollback budget). Structured payloads are
JSON; hot-path payloads are raw bytes. PROTO_VERSION = 5, checked at
adoption time from Hello.version — a version-skewed worker is refused, not
guessed at.
| Type | Dir | Payload | Purpose |
|---|---|---|---|
T_HELLO 0x01 | w→d | JSON {version, session_id, instance_id, state, pid?, cols, rows, cwd?, history?} | first frame on every accepted connection; enables stateless adoption, binds lifecycle to this exact worker instance, and in v5 retains the canonical absolute cwd capability required by direct session uploads. history: true advertises the committed-history delta stream (§8.4). Decoders accept the pre-v3 agent_id spelling as an alias so a new spawnd adopts workers started before the rename |
T_START 0x02 | d→w | JSON {cwd, argv, env, cols, rows} | spawn the session command (spawnd always sends the user's login shell with -l). Env goes over the private socket, not argv, so secrets never appear in /proc/*/cmdline |
T_STARTED 0x03 | w→d | JSON {pid, cwd} | session is running; cwd is the canonical absolute capability root retained by the worker |
T_OUTPUT 0x04 | w→d | watermark u64 LE ‖ raw bytes | live PTY output with the same durable producer coordinate used by replay |
T_INPUT 0x05 | d→w | raw bytes | PTY stdin |
T_RESIZE 0x06 | d→w | cols u16 LE, rows u16 LE | PTY resize (kernel sends SIGWINCH); lines displaced by a narrowing reflow commit to history |
T_REDRAW 0x07 | d→w | empty | obsolete (ignored by workers; reserved — see §8.2) |
T_REPLAY_REQ 0x08 | d→w | max_bytes u32 LE | request decrypted scrollback |
T_REPLAY 0x09 | w→d | watermark u64 LE ‖ raw bytes | self-describing v2 replay: geometry marker + history sentinel + committed lines, then geometry marker + live screen repaint (§8.1); watermark = cumulative lifetime PTY output bytes at capture, including output never committed to history |
T_EXIT 0x0A | w→d | JSON {exit_code?, signal?} | session exited |
T_SHUTDOWN 0x0B | d→w | JSON {signal?: TERM|KILL} | compatibility command; current spawnd lifecycle delivery uses the independent endpoint below |
T_ERROR 0x0C | w→d | JSON {message} | recoverable command failure |
T_HISTORY 0x0D | w→d | epoch u64 LE ‖ start_offset u64 LE ‖ committed bytes | one committed-line batch, streamed live as it is persisted (§8.4). Only sent after T_HISTORY_SUB |
T_HISTORY_WIPE 0x0E | w→d | epoch u64 LE | the app erased its scrollback (ED 3); carries the new epoch, offsets restart at 0. Only sent after T_HISTORY_SUB |
T_REPLAY2 0x0F | w→d | watermark u64 LE ‖ epoch u64 LE ‖ history_end_offset u64 LE ‖ raw bytes | replay response with the committed-history anchor at capture; replaces T_REPLAY for subscribed supervisors. The end offset always lands on a batch boundary, so a delta starting there appends seamlessly |
T_HISTORY_SUB 0x10 | d→w | empty | subscribe to the delta stream. Sent only to workers whose Hello advertises history: true; a worker never emits the three frames above unsubscribed, because supervisors reject unknown frame types by dropping the connection |
T_FOREGROUND 0x11 | w→d | raw UTF-8 basename, ≤ 256 bytes | foreground process report (§4.1). Sent when the polled value changes and once per new supervisor connection. Additive at PROTO_VERSION 5: old workers never send it |
Connection semantics: the worker serves one live supervisor connection.
It validates the candidate peer's effective UID on Unix or process-token user
SID on Windows and sends that candidate its
Hello; only then does it close the old writer, abort and await the old reader,
and install one new reader. Thus only one task/fd can feed the bounded command
queue. Generation tags also discard anything the old peer queued immediately
before cancellation. A restarted spawnd connects and wins without leaving a
stale reader able to flood the worker. spawnd verifies that
Hello.session_id matches the session implied by the endpoint before trusting
the instance token. Unknown frame types are rejected. The five-byte header is
parsed before allocation and a strict per-type cap is applied (T_INPUT is at
most 64 KiB; fixed commands require their exact size); a command never inherits
the generic 32 MiB replay ceiling.
Lifecycle timers: a worker that never receives Start exits after 120 s; a
worker whose session exited lingers 60 s to deliver T_EXIT to a reconnecting
spawnd, then cleans up regardless. On exit the worker deletes its scrollback
(the key dies with it anyway), identity-checks its endpoint metadata, and
terminates.
The lifecycle endpoint is separate and adoptable, not another command in the
ordinary frame queue. A request is exactly 17 bytes: the 16-byte random
Hello.instance_id plus a one-byte TERM/KILL enum. Unix sends one atomic
datagram and serves it with one task plus one fixed 18-byte buffer. Windows
sends one message on the -lc pipe and admits at most seven fixed-buffer
handlers; a connected client that writes nothing expires after 100 ms. Both
clients retry idempotently within one absolute two-second deadline and reject
stale instances, unknown codes, malformed lengths, and malformed one-byte
acknowledgements.
The worker retains the unreaped portable-pty Child handle as stable process
identity. Unix validates it under the exit-monitor lock and calls killpg
only; ESRCH is safely gone, with no bare-PID fallback. Windows TERM is an ETX
(Ctrl+C) priority write to ConPTY input, independent of the ordinary bounded
input queue, and may be cooperatively ignored. Windows KILL flushes the
lifecycle acknowledgement first and then terminates the worker Job Object,
removing the worker and its complete descendant tree.
4.1 Foreground reporting (T_FOREGROUND)
On Unix, once per second while its session is running, the worker asks the kernel
which process group owns the PTY foreground — tcgetpgrp on the PTY master
fd — and resolves that group's leader to an executable basename:
/proc/<pgid>/comm on Linux, libproc proc_name (falling back to
proc_pidpath) on macOS (sessiond/foreground.rs). The result is sent as
T_FOREGROUND only when it changes, and re-sent once to each newly accepted
supervisor connection so an adopting spawnd learns the current value without
waiting for a change.
spawnd relays it to the server as session.foreground — deduplicated by
value, rate-limited to one frame per second per session, and re-announced when
the control websocket reconnects. The payload is deliberately minimal: the
basename only, truncated to 64 characters — no arguments, no paths, no
output, no titles. This is a documented content-free-design exception
(docs/TRUST.md); it exists so the UI can label panes and decide when the
shell is in the foreground. Old workers (pre-rename) simply never send the
frame; their sessions report no foreground command.
ConPTY has no public tcgetpgrp equivalent. Windows therefore uses an
advisory Toolhelp snapshot rooted at the retained START-process PID: it walks
the descendant tree and reports the newest live descendant's executable
basename (falling back to the root). PID reuse, snapshot races, and background
descendants can produce false positives; this is display metadata, never an
authorization or lifecycle identity.
5. Data path — where bytes flow, who can read them
session process (login shell + whatever it runs)
│ PTY slave → kernel → PTY master (plaintext, kernel buffers, user's host)
▼
spawn-worker: read buffer ── encrypt → scrollback log (ciphertext, disk)
│ └─ zeroized after each hop
▼ T_OUTPUT (protected Unix socket / local named pipe, same host)
spawnd: bounded per-session outbox → forwarder ──→ DataChannel direct sinks
▼ WebRTC DataChannel (DTLS, peer-to-peer; TURN sees ciphertext)
browser: xterm.js — the user-facing terminal renderer and scrollback owner
pty::run_forwarder and ForwarderControl provide bounded outbox → direct-sink
routing. SessionHandle has one implementation: write_stdin, resize, and
replay dispatch bounded WorkerCmds over the worker endpoint. Shutdown binds a
short-lived 0600 Unix datagram endpoint or connects to the bounded Windows
lifecycle pipe and sends to the separate worker-owned endpoint with the instance ID captured from
the same Hello; one absolute deadline covers validation, fixed-size delivery,
retries, and acknowledgement. It therefore cannot sit behind queued or
partially written PTY input. Registry delivery also revalidates the immutable
generation+lifecycle pair while holding the generation-transition lock.
Restart checks TERM delivery and deterministically escalates to KILL. spawnd
does not hold a local session PTY, a child process handle, or a backend
discriminator.
The protected same-user local hop adds no control-plane exposure. P2-AGENT-02 removed daemon
WS terminal binary frames, browser relay/history/snapshot frames, transcripts,
and content pubsub. spawn.ctl replay carries a stream-position watermark
directly to the browser. Session uploads now use that direct channel, and host
file operations use spawn.host.ctl. The broader Phase 2 claim remains
incomplete because session directories, agent-definition values, skill bodies,
host-agent operations, some detailed errors, and historical purge still have
server-readable or unfinished paths; see TRUST.md.
6. Encrypted-at-rest scrollback
daemon/src/sessiond/scrollback.rs.
6.1 Format
Append-only, segmented log. Per record:
u32 LE ciphertext_len | u8 kind | u64 LE seq | ciphertext (AEAD, 16-byte tag)
- Cipher: ChaCha20-Poly1305, one AEAD seal per record.
- Nonce: the strictly-monotonic record sequence number (96-bit nonce, low 64 bits = seq). Safe because the key is unique per worker process and never reused (§7); nonce reuse is structurally impossible.
- AAD binds
kind ‖ seq, and replay verifies seq contiguity across records, so ciphertext records cannot be reordered, dropped, duplicated, or spliced between kinds without detection. Any authentication or sequence failure fails the whole replay closed (and zeroizes the partial plaintext) rather than returning a best-effort screen. - Kinds:
HISTORY(a batch of committed scrollback lines, serialized as self-contained styled text; §8.1). An app-driven scrollback wipe (ED 3) is not a record at all — it physically unlinks every retained segment (truncate_all), so cleared history stops existing on disk. The seq counter never resets across truncation, so nonces cannot repeat.
6.2 Encrypt-on-read, bounded growth
Each output chunk is fed into the emulator; the lines it commits are
encrypted before any scrollback write. The worker subsequently forwards the
same plaintext chunk live and wipes its owned buffers. This is an
encrypt-before-disk property, not encryption at the PTY/DRAM boundary.
Plaintext is never written to segment files (unit-tested by grepping them for
a marker; plaintext_never_hits_disk).
Growth is controlled by two knobs. --segment-bytes defaults to 256 KiB of
additional charged record bytes per segment before rotation
is due. --max-log-bytes defaults to an 8 MiB conservative total scrollback
resource budget. Operators/tests may lower that value; values above the
compiled 8 MiB upper bound are rejected. The charge includes exact retained
ciphertext and record framing, twice each segment's replay representation (one
returned buffer plus one decryption/framing scratch allowance), and the log's
retained Vec/path bookkeeping, actual allocated file/directory blocks with
safe floors, and conservative inode/directory-entry overhead. A hard 128-file
cap plus ring-reused filenames bounds live inodes and directory growth under
one-byte-output/resize adversaries. Before admitting a record the log removes
whole oldest segments; it never returns a partial segment. A record that
fails conservative preflight is rejected before its file is created. If an
append or rotation cannot preserve those invariants, the worker destroys and
disables its replay log but continues live output; subsequent replay is
unavailable rather than partial or over-budget.
The 8 MiB value is therefore neither "8 MiB of plaintext output" nor an exact
measurement of process RSS. It is a hard ceiling on this deliberately
conservative charge model, including grid/line serialization in replay
form and bounded transient replay plaintext. The live emulator grid is a
separate, geometry-bounded resident allocation (§6.3). The spawn.ctl layer
also retains its independent 12 MiB response rejection ceiling.
6.3 Memory hygiene — what is and isn't guaranteed
daemon/src/sessiond/secret.rs (SecretBytes):
Covered:
- Key bytes are generated from
getrandom, held in heap pages that aremlock(2)ed on Unix or best-effortVirtualLocked within the Windows working-set limit,MADV_DONTDUMPed on Linux (no core dumps), and zeroized on drop. mlock failure (e.g.RLIMIT_MEMLOCK=0containers) is logged, not fatal: the at-rest encryption stands; only the key's swap-residency guarantee weakens. Windows has no per-allocationMADV_DONTDUMPequivalent, so no dump-exclusion claim is made there. - Owned plaintext is explicitly wiped on drop across the implemented handoff:
worker PTY read chunks and reader/writer scratch, queued input and worker
frame payloads, serialized committed lines, and worker replay buffers. spawnd's
replay result owns a self-wiping payload even while parked in a oneshot; its
source bytes wipe on receiver cancellation and normal consumption.
DataChannel input is copied into the self-wiping worker-input wrapper.
OutputChunk, control output, and direct-viewer payloads likewise wipe on drop; their queues are bounded.
Not covered — stated plainly, per TRUST.md's "honest inventory" ethos:
- Plaintext must transit worker memory: kernel PTY buffers → userspace read buffer → AEAD input. There is no such thing as "encrypted before DRAM" on this path, and we do not claim it.
- The headless emulator retains semantic plaintext for the current primary and alternate screen, cursor, modes, and auxiliary terminal state for the worker's lifetime. That state scales with terminal geometry and has no deep history, but it is not transient. A serialized line batch and decrypted replay are additional transient plaintext buffers. The scrollback admission charge accounts conservatively for retained disk records, the returned replay, and decryption/framing scratch within its 8 MiB default; the control-channel response ceiling is an independent outer bound.
- Kernel-side copies (PTY line discipline, local endpoint buffers) and copies inside webrtc/DTLS layers in spawnd are outside our control.
- spawnd still handles plaintext in flight (worker socket → bounded outbox → DataChannel). The outbox, worker-command channel, direct-viewer queues, and control responses are bounded; a lagging direct viewer is detached, and reconnect catch-up comes from worker replay. Owned queued payloads wipe on drop. Copies inside kernel endpoint/WebRTC/DTLS stacks are not owned or wiped by this code, so this is not a claim of complete system-wide zeroization.
- The guarantee here is ciphertext-only scrollback segment files on the user's own host, with best-effort wiping of transient buffers — not the absence of plaintext host memory. The threat this addresses is disk residue (backups, stolen disks, forensic carving, swap, core dumps), not a live root-level attacker on the host, which TRUST.md places out of scope (a compromised host sees everything regardless).
7. Key management for scrollback-at-rest
Implemented: per-worker-process ephemeral key. 32 random bytes generated at worker start, held only in locked worker memory, never persisted, dead with the process. A fresh worker unlinks any leftover segments from a previous run (unreadable ciphertext by construction).
Why this is the right default — the alternatives and their tradeoffs:
| Key scheme | Survives worker restart | Blast radius if key leaks | Server involvement | Notes |
|---|---|---|---|---|
| Per-worker ephemeral (chosen) | no — but the PTY and session process tree died with the worker anyway, so the log has nothing meaningful left to replay | one current session | none | zero key-management surface; nonce safety trivial; cleanup = forget |
| Host key (spawnd keyring) | yes | every session's scrollback on the host | none | requires spawnd→worker key delivery (over the socket, fine) and rotation story; buys persistence of logs whose PTY is gone — mostly useful for a future "transcript archive" feature, not live reattach |
| Per-session derived key (HKDF from host key + session ID) | yes | one session's full history | none | same delivery/rotation cost as host key with a smaller blast radius; the natural upgrade path if worker-restart-with-history ever becomes a feature |
| Sealed to browser device keys (Phase-3 WebCrypto identities) | yes | nothing on the host can read it — including the worker | pub-key registry only | the strongest story ("host stores what only your devices can open") but the worker could no longer serve replay; replay/serialization logic would move client-side, multi-device needs key-wrapping fan-out. This is the TRUST.md "client-side-encrypted transcript backup" (Later) item, not the live-session log |
The live-session log exists to serve reattach while the session is alive; the ephemeral key covers exactly that lifetime with the smallest possible surface. When TRUST.md's optional encrypted transcript backup lands, it should be a separate artifact sealed to device keys, not a repurposing of this log.
8. Reattach and replay
8.1 Committed-line history
History is owned by the emulator, not reconstructed from bytes. A line is
committed exactly once, at the moment it scrolls off the top of the
screen (alacritty's grid provides the semantics: full-screen scrolls and
top-anchored regions rotate lines into grid history; ED 2 scrolls the
viewport into history, VTE/kitty-style). After every feed stride the worker
drains those lines — serialized as self-contained styled text: SGR runs +
glyphs, \r\n per hard line end, soft-wrapped rows painted edge-to-edge with
no break so logical lines re-wrap at the consumer's width — and appends them
encrypted (HISTORY records). In-place TUI repaints never scroll, so they
never commit; a resize cannot retroactively reflow committed lines (narrowing
commits the displaced rows once, widening finds an empty drain window and has
nothing to un-commit); ED 3 (/clear in claude/codex emits 2J 3J H)
physically truncates the log (scrollback_wipe_erases_replayed_history).
Segment rotation is purely a storage/eviction concern — the session process
is never signaled, resized, or otherwise disturbed by it
(scrollback_rotation_must_not_disturb_the_agent; the test name is retained
for compatibility history).
Replay concatenates the newest run of whole segments whose plaintext fits the
caller's max_bytes (failing closed rather than splitting a segment), then
the worker frames the response as a self-describing v2 stream:
CSI 8 ; rows ; cols t APC "sp:h1" ST ESC ( B ESC ) B ESC * B ESC + B SI <committed lines…>
CSI 8 ; rows ; cols t <emulator-serialized live screen repaint>
Both markers carry the current geometry; the history section is
geometry-free flowing text. The screen repaint is synthesized from the live
emulator at request time (idempotent: full-row painting, no ED), so the
final chunk alone reconstructs the current screen — the browser's live
terminal is still seeded from the final chunk with zero resize calls.
Both sections first return the consumer to ASCII — the head right after
the sentinel (scrollback::replay_head), the repaint in its baseline
(emulator::RETURN_TO_ASCII in both): every glyph the worker paints is
already mapped through the app's character sets, and a consumer the app's
live bytes or the previous repaint's tail left in a line-drawing set would
map it a second time (#61). The repaint's tail then re-arms the app's
designations and shift state, so its next live bytes render alike in the
worker and the consumer. The clients' reseed clear leads with the G0, G1
and SI part of the same return; the stream does not depend on it. Live
history_delta batches carry no return of their own (nothing consumes
them yet): a client that appends them into a terminal the app is drawing
on must return it to ASCII first and re-arm the app's sets after.
The scrollback overlay, on recognizing the APC sentinel
(parseHistoryReplay in Terminal.tsx), writes the history as flowing text
at its own width (never geometry-walked, so nothing already rendered ever
reflows), scrolls the occupied viewport rows into the scrollback region, and
paints the screen chunk below. Clients without the sentinel fall back to the
legacy chunk walk; replays from pre-v2 workers (which persist across
upgrades) still parse via the same marker framing.
Two prior designs were retired. Checkpoint markers + SIGWINCH jiggle duplicated full frames in scrollback on every rotation and left replay quality dependent on app WINCH behavior. Its replacement — raw-byte segments opened by emulator-serialized checkpoints — fixed the jiggle but kept re-executing the byte stream to rebuild history, which pushed intermediate repaint frames into the overlay's scrollback and reflowed (mangled) TUI rows at every geometry transition. Committed lines fix the class: history is a document written once, not a render re-run.
8.2 The emulator
sessiond/emulator.rs wraps alacritty_terminal's Term (grid history is
enabled but used only as a bounded drain window — feed_output serializes
and clears it every stride; deep history lives in the encrypted line log)
plus a shadow handler on a second vte parser for the states Term keeps
private (margins, the SI/SO shift state, and the shift each screen's DECSC
register was saved under — alacritty's register has no room for it, and an
xterm.js consumer's register holds the table that shift selected; the
G0–G3 designations are read from the grid cursor, where alacritty keeps
them per screen and DECSC/DECRC save and restore them). serialize()
emits an ANSI stream
reconstructing cells, attributes, hyperlinks, wide/combining chars, cursor
(including pending wrap), margins, modes, charsets, cursor style, and palette
overrides — for both screens when the alternate screen is active — including
the DECSC saved-cursor register, which Ink renderers (claude, codex) rely on
around every frame. A byte-exact cross-chunk scanner detects ED 3
(CSI 3 J / CSI ? 3 J) and surfaces it as a truncate event ordered into
the commit stream. The fidelity contract (feed → serialize → re-feed ⇒ identical state) is enforced cell-by-cell by the module's unit tests, and
end-to-end by replay_reconstructs_the_live_screen_across_rotations, which
renders the live byte stream and the replay through two emulators and
requires identical
screens. T_REDRAW is obsolete and ignored by workers: snapshots synthesized
from the emulator already carry cursor and modes, so there is nothing left to
provoke.
8.3 Replay and viewer seeding
T_REPLAY_REQ(max_bytes) → T_REPLAY(watermark ‖ bytes). The watermark is
the cumulative lifetime count of output bytes logged at capture time; it is a
monotonic source coordinate, not the size of the retained replay tail. spawnd's
worker-backed replay path samples the
requesting viewer's DataChannel byte offset before issuing the replay;
because the worker logs before forwarding, everything counted at that offset
is guaranteed to be covered by the replay, and the browser can drop already
seen live bytes deterministically using dc_offset.
spawn.ctl history/snapshot exposes this as styled terminal replay only:
clients send plain:false. A plain:true request fails closed with
plain_replay_unsupported; ANSI replay bytes are never mislabeled
as plain text.
Replay correctness (PTY in → bytes out → reattach replays both the initial
output and mid-session stdin echo) is asserted end-to-end in
daemon/tests/worker_e2e.rs against a real /bin/sh on a real PTY, and
through the full spawnd plumbing (forwarder, direct sinks, adopt path) in
worker_backend::tests::worker_launch_adopt_and_priority_shutdown_roundtrip.
8.4 Live committed-history deltas
Replay seeds the scrollback view; deltas keep it exact forever after. The
worker streams every persisted history effect to its supervisor as it happens
— T_HISTORY per committed batch, T_HISTORY_WIPE per ED 3 — and spawnd
fans them out per viewer as spawn.ctl events (history_delta with a base64
fragment, history_wipe, and history_gap when a viewer's bounded queue
overflowed and it must re-anchor). The browser's scrollback overlay is
therefore a pure view of the worker's log: seeded from a replay's history
section, extended only by deltas, with raw live PTY bytes never entering its
buffer. The live screen is painted below the history at reveal time from the
local live terminal (client-side serialize), so the seam cannot interleave and
nothing is fetched to open scrollback.
Continuity is a two-part anchor: epoch (a per-worker-process nonce, bumped
on every wipe, sent as a decimal string on spawn.ctl because u64 exceeds
JavaScript safe integers) and offset (committed plaintext bytes within the
epoch). Replay anchors always land on batch boundaries; ctl-level fragments
split batches at byte granularity and re-chain by offset. Any hole, epoch
change, or overflow makes the client drop its anchor and heal with exactly one
fresh snapshot. Everything is capability-gated end to end: workers advertise
history: true in Hello, daemons subscribe with T_HISTORY_SUB, clients
opt in with a history_subscribe ctl request — so old daemons, old workers,
and old (cached) clients all continue on the legacy replay-only flow with no
frame-type surprises.
9. Resize, flow control, multi-viewer
Resize. SessionHandle::resize dedupes unchanged geometry and
sends T_RESIZE; the worker applies it to the PTY master and the emulator,
committing any lines a narrowing reflow displaces. Resize authority is
negotiated endpoint-to-endpoint over spawn.ctl: the daemon's display-control
hub designates one
controlling viewer whose geometry drives the session while other viewers dim
without sending geometry or viewer timing through the application server. The
worker therefore stays a single-size PTY and needs no multi-size machinery.
Multi-viewer. Fan-out happens in spawnd's forwarder via per-viewer DataChannel direct sinks: every viewer gets the same raw byte stream, and input is accepted from whichever peer the control model lets type. There is no daemon-side copy-mode or per-keystroke subprocess check. Client-side scrollback/selection lives in xterm.js.
Flow control / backpressure — current, honest status. The PTY reader's
handoff to the worker loop holds at most eight queued chunks of at most 8 KiB
each. The worker logs and forwards each chunk it consumes. If the supervisor
endpoint stalls, those eight slots fill, the PTY reader blocks, and the kernel
PTY backpressures the session instead of accumulating an unbounded worker Vec
queue. Downstream, spawnd holds at most 32 worker-output chunks and 32 ordinary
worker commands; each input command is capped at 64 KiB before spawnd copies
the caller's slice. The worker's separate lifecycle endpoint is independent of
the ordinary command task: Unix accepts one atomic request into one fixed
buffer, while Windows caps message-pipe handlers at seven and expires silent
clients after 100 ms. TERM/KILL therefore remain bounded when the ordinary
queue or supervisor endpoint stalls. The forwarder serves
only bounded direct-viewer sinks; a lagging viewer is disconnected instead of
accumulating plaintext. A new direct connection re-seeds from the worker replay
watermark. The replay log uses the conservative total resource budget
described in §6.2, including checkpoint and framing charges.
10. Crash isolation, restart, upgrades
| Event | Outcome |
|---|---|
| Session exits | worker reports T_EXIT (real exit code), destroys its scrollback, identity-checks and removes Unix sockets or the Windows marker, releases its lifetime reservation, and exits; spawnd forwards session.exit. If spawnd is down at that moment, the worker lingers 60 s so a restarted spawnd can collect the exit. |
| Worker crashes | the session process tree dies with it (PTY ownership on Unix; kill-on-close Job Object on Windows) — identical blast radius to "tmux server crashed" but scoped to one session instead of every session on the host. spawnd's connection reader reports worker_lost. The kernel releases the flock/exclusive HANDLE; the next launch, or a failed adoption that acquires that reservation, ownership-checks and removes stale endpoint metadata. Restart policy is user-driven session.restart; blind auto-respawn of stateful work inside the shell is not a recovery. |
| spawnd restarts / upgrades | workers keep running in independent Unix process groups or detached Windows breakaway trees. On startup rediscover_existing_sessions scans protected .sock entries or .endpoint markers via discover_ids, connects, verifies Hello.session_id, and adopts from Hello (instance identity, state, pid, geometry, cwd) — no persistent supervisor state. The same lazy path (ensure_session_attached) recovers a session on first use if startup discovery raced. A reconnect displaces no session state; viewers re-seed from emulator-synthesized snapshots on demand. Hello accepts the former identity-key spelling as an adoption-only serde alias. |
| spawnd upgrade + protocol change | Hello.version gates adoption; private worker protocol version 5 additionally requires the worker's retained canonical cwd capability root for direct session uploads. A mismatched or cwd-less worker is left untouched (its session keeps running) and surfaced in logs rather than driven with a protocol it does not speak or falling back to a server upload path. Old workers drain away as their sessions exit. |
| Worker binary upgrade | applies to newly launched sessions only; running workers are never hot-swapped. worker_bin() resolves $SPAWND_WORKER_BIN → sibling of the running spawnd binary → PATH, using the platform executable suffix. |
| Host reboot | workers and their session process trees die. Unix runtime-dir sockets/ciphertext evaporate with tmpfs; Windows named pipes disappear and stale protected markers are recoverable under the released reservation. |
On Unix, the only launch-time fd inheritance is the already-flocked
reservation, exposed to that one post-fork child and immediately restored to
close-on-exec. On Windows, STARTUPINFOEXW and an explicit
PROC_THREAD_ATTRIBUTE_HANDLE_LIST transfer only the exclusive reservation
and three NUL standard handles; the worker clears reservation inheritance and
logs to its protected worker.log. PTY and endpoint handles are never handed
off: the worker owns them and survives on its own. Adoption-by-reconnect has
no listener handoff window.
11. Worker-only cutover
P2-TMUX-01 is the hard cutover, not a default flip. The daemon module, subprocess calls, selector and escape hatch, creation/attach/discovery, capture/repaint/copy-mode behavior, session label, and exact replay buffer are removed. Session restart performs escalating TERM→KILL through the worker before creating a replacement; startup discovery scans worker sockets only.
A live pre-cutover session cannot be adopted or transformed into a worker without unsafe content/state handling. Operators close ingress and drain it before installing/restarting the worker-only build. If one remains, the daemon fails it unavailable. This code change itself performs no deployment, signal, or purge. Rollback cannot restore the retired content path; remediation rolls forward with a corrected worker-only build. See TMUX_REMOVAL.md.
The server/API/web tmux_session field and agent.rename frame are removed in
the same checkpoint. They are absent from daemon runtime structs as well.
12. Prior art
| shpool (Google) | wezterm-mux-server | zellij server | spawn-worker | |
|---|---|---|---|---|
| Language | Rust | Rust | Rust | Rust |
| Model | one daemon, N named sessions | one mux server, own client protocol | one server per session group, plugin runtime | one worker process per session |
| Server-side emulation | minimal (keeps a restore buffer; explicitly not a multiplexer) | full (termwiz grid; clients render grid deltas) | full (its own grid + layout engine) | headless checkpoint emulator only (alacritty core, plaintext current-screen grids); live path is raw bytes, user-facing rendering is xterm.js |
| Reattach story | replays restore buffer | grid sync | grid sync | encrypted geometry-tagged byte replay opening with an emulator-serialized checkpoint (the iTerm2 restoration model, encrypted at rest) |
| Scrollback at rest | plaintext in memory | plaintext (grid) | plaintext (grid) | ChaCha20-Poly1305 on disk, ephemeral key; transient owned replay buffers wiped, plaintext current grids disclosed above |
| Crash blast radius | all sessions in daemon | all clients of the mux | session group | one session |
| Remote transport | ssh | ssh/TLS, own protocol | ssh | WebRTC DataChannel (already existed; unchanged) |
shpool is the closest relative — it also concluded that "session persistence without a multiplexer" is the right shape, and its restore-buffer replay is the plaintext cousin of our checkpoint replay. We diverge where the operator model demands it: per-session process isolation, encryption at rest, and keeping the worker's plaintext grid state to the current checkpoint screens rather than a user-facing or deep-history multiplexer. wezterm/zellij solve a different problem (rich multiplexing UX) with a server-side renderer in the live path — exactly what §1 is eliminating.
13. Testability
The browser is the only user-facing terminal renderer, while the worker has a headless checkpoint emulator (§2). Terminal-correctness testing therefore splits into two independently-testable layers:
Byte-transport correctness (this design). The worker/backend guarantee is
byte-exactness, tested without any emulator: wire framing round-trips and
limits (sessiond::wire tests), crypto/log properties — plaintext never on
disk, tamper fails closed, budget enforcement, replay coherence
(sessiond::scrollback tests), memory hygiene (sessiond::secret tests),
and end-to-end PTY-in/bytes-out/reattach-replay through a real shell
(daemon/tests/worker_e2e.rs) and through the full spawnd plumbing
(worker_backend roundtrip test). cargo test in daemon/ runs all of it
without network or browser dependencies.
Emulation correctness (harness). tools/term-conformance/ owns this:
a raw-byte corpus is fed to the system-under-test emulator (@xterm/headless
5.5.0, matching web/) and to oracles (pyte locally; iTerm2 recorded on
macOS), and resulting screen states are diffed.
Grid-state schema — ratified. This design adopts the harness's
grid-state JSON schema v1 (tools/term-conformance/schema/ grid-state.schema.json, documented in schema.md; proposed in the shared
log 2026-07-13T11:02Z) as the serialization of terminal screen state for
all conformance and replay-fidelity testing, as proposed — including its
0-based coordinates, default-omitting compact cells, wide-char continuation
cells (width: 0), "default" | 0–255 | "#rrggbb" colors, NFC-normalized
text comparison, cursor-col clamping for pending-wrap, and null-as-wildcard
for limited oracles (that last rule is what lets a raw-byte-replay producer
that cannot observe, say, the title report null without false failures).
The schema is also the bridge between the two layers: because worker replay
is raw bytes, replay fidelity is testable by feeding (a) the live stream
and (b) a post-rotation replay of the same session into two headless xterm
instances and diffing their grid-state JSON — "replay from a checkpoint
converges to the live screen" becomes a machine-checkable property using the
harness's differ unmodified. A Rust-side version of that property already
runs in worker_e2e.rs (replay_reconstructs_the_live_screen_across_ rotations, diffing through the sessiond emulator); the xterm.js-side
integration is future work in tools/term-conformance territory and should
reuse its runner. The checkpoint emulator itself is a natural additional SUT
for the corpus: xterm.js(serialize(emulator(case))) ≡ xterm.js(case).
14. TRUST.md phase mapping
| TRUST.md phase | sessiond contribution |
|---|---|
| Phase 1 (shipped) — DataChannel-only PTY | worker backend removed tmux as a plaintext holder; P2-AGENT-02 removed the legacy server mirror |
| Phase 2 — endpoint-owned data, server stores deleted | scrollback/history/snapshot and session uploads are worker-owned and use spawn.ctl; host file operations use spawn.host.ctl. Session directories, agent-definition values, skill bodies, host-agent operations, some detailed errors, and historical purge still prevent a complete Phase 2 claim |
| Phase 3 — signed signaling | orthogonal to sessiond (signaling-layer); nothing here assumes server-trusted introductions |
| Phase 4 / Later — open source; encrypted transcript backup | worker is self-contained and auditable (daemon/src/sessiond/ has no control-plane deps); device-key-sealed backup slots in as a separate artifact per §7 |
15. Known gaps / future work
- Backpressure follow-up (§9): Phase 2 uses bounded per-viewer sinks, disconnect-on-stall and replay-based catch-up; future work may add adaptive queue sizing and transport telemetry.
- Replay-fidelity conformance test (§13): grid-diff live vs replayed streams via the harness.
- Worker resource limits: per-worker RLIMIT/cgroup knobs if session workloads start sharing hosts with untrusted workloads (out of scope for the current single-user host model).
spawnd statusintegration: list worker-backed sessions with pid/state fromHellowithout attaching.