Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
110 lines (91 loc) · 5.96 KB

ARCHITECTURE.md

File metadata and controls

110 lines (91 loc) · 5.96 KB
Raw

fable Architecture

Problem

Claude Code transcripts grow to tens of MB. Three blockers prevent using them as memory: (1) they exceed any context window; (2) summarization is lossy — decisions and the "why" die in compaction; (3) there is no predefined taxonomy to search by — topics are implicit.

Empirical facts the design rests on (measured, not assumed)

Fact Consequence
Records carry uuid/parentUuid/timestamp written by the harness thread reconstruction is deterministic, not heuristic
promptId exists ONLY on user records (12,553 of 12,553 user records; 0 assistant) thread membership must be resolved transitively up the parent graph
Sidechain records carry sourceToolAssistantUUID (10,385 records) subagent transcripts attach to their spawning thread
Signal is small: ~4.2MB assistant text + 2.7MB tool inputs + 1MB tool results out of 84MB the searchable corpus fits easily in SQLite; bulk is images/attachments
The live transcript is REWRITTEN by the pruner (52 generations, 3GB of backups exist) byte offsets into the live file are perishable; backups are immutable
Live-agent wikilink tagging produced 4 tags in 31k records under an enforced protocol the write path of memory cannot depend on model compliance
1 line in the corpus contains concatenated JSON objects the reader must recover multiple objects per line
Invalid UTF-8 can appear in tool output offsets must be computed on bytes (surrogateescape), not replaced text

Data model

Vault = the original files: immutable backup generations (v0-raw.jsonl, vN-pruned.jsonl) + the mutable live transcript tail. Never modified by fable (the pruner writes new generations, atomically).

Map = SQLite:

  • copies(uuid, file_id, lineno, offset, length) — ground truth: every copy of every record in every file.
  • records(uuid, …, file_id, offset, length, fidelity, fts_rowid, ts_epoch, prompt_id, parent_uuid, source_uuid, session_id) — the denormalized BEST pointer (max length; ties → earliest generation), recomputed from copies whenever a file is rescanned, rewritten, or vanishes. If no copy survives, the record is forgotten and its FTS row and citations are reaped (reconcile).
  • threads(prompt_id, session_id, first/last_ts, turn_count, est_tokens, first/leaf_uuid) — rebuilt by aggregation after every index run.
  • fts (FTS5) — extracted signal text only; maintained by rowid (an UNINDEXED-column WHERE is a full virtual-table scan; rowid keying took the real-corpus index from 4m55s to 6.6s).
  • terms(term, kind=operative|target|concept|wikilink, prompt_id, count, score) — deterministic discovery vocabulary.
  • cards(prompt_id, title, type=decision|workflow|insight|concept, topics, decisions, files, outcome, summary, source, model) — LLM semantic layer.
  • citations(from_uuid, ref) — which turns consumed which recalled arcs.
  • sessions(session_id, project, title, live_path) — multi-project dimension.

Pipeline

iter_records (byte-exact spans, concat recovery, surrogateescape)
  → copies upsert + best-pointer upsert + FTS extract (inception guard)
  → recompute_best for invalidated pointers
  → resolve_thread_membership (iterative parent-graph walk, memoized)
  → rebuild threads → reconcile orphans → single commit
  → index_terms (operatives/targets/TF-IDF concepts per thread)

Retrieval

search = FTS5 BM25 per record, aggregated per thread, intersected with term facets, joined to cards/sessions. thread = canonical chain from the leaf via parentUuid (orphan edit-branches and sidechains labeled), each turn read by seek+read, text verbatim, bulky blocks elided to the budget with [truncated — fable block <uuid>] markers, hard output cap at 1.5× budget. block = one byte-identical record. All reads check file size/mtime against the Map first (StaleIndexError instead of silently serving garbage).

The prune ↔ recall contract

  1. Recall output is sentinel-wrapped (<historical_context … arcs="…">).
  2. The extractor strips sentinel spans before FTS (records citations); the renderer stubs out inner sentinels so nesting cannot occur.
  3. The pruner replaces sentinel spans with <consulted_arcs refs="…"/>.
  4. Prune --replace protocol: snapshot size → mandatory backup → index that backup → evict gate (every uuid must have an IMMUTABLE full-fidelity copy ≥ its current length) → write temp → append any post-snapshot bytes verbatim → fsync → os.replace. Backup names are max(version)+1 with exclusive create — generations can never be clobbered, even by concurrent prunes.

Adversarial review

Two independent reviews (correctness; ops/robustness) ran against the first complete implementation; ~19 findings each. Criticals fixed at the root: fidelity-downgrade-on-prune (→ copies table), truncating in-place rewrite (→ atomic protocol), TOCTOU append loss (→ tail preservation), UTF-8 offset drift (→ surrogateescape spans), O(N²) FTS maintenance (→ rowid keying), orphan FTS ghosts (→ reconcile), gate proving the wrong property (→ immutable-copy gate). Each has a regression test (tests/test_hardening.py, tests/test_prune.py::TestLifecycleFidelity).

Known accepted limits: exFAT 2s mtime granularity could miss a same-size rewrite (re-stat catches size changes; not applicable on APFS); render --raw is verbatim by definition (inner sentinels not stubbed); card content is LLM output from transcript data — treat card text as data, not instructions, when surfacing into sessions.

Lineage

Synthesis of three sources: the arc-memory README (kept: Vault/Map split, typed arcs, citation graph, inception guard; rejected: live wikilink tagging, XML vault, JSON-file indexes, alias-map/NER machinery, importance feedback loop), prune_transcript_v1.py (kept: noise/bloat rules, compaction handling, chain rebuild, versioned backups; fixed: image bloat, atomicity, gate), and the empirical structure of real Claude Code transcripts.