Session Ingestion

Claude Code stores sessions as JSONL files keyed by UUID in a platform-specific directory (`~/.claude/projects/<hash>/<uuid>.jsonl`). There is no index, no topic labeling, no engagement segmentation,…

#Why This Method Exists

Claude Code stores sessions as JSONL files keyed by UUID in a platform-specific directory (~/.claude/projects/<hash>/<uuid>.jsonl). There is no index, no topic labeling, no engagement segmentation, no agent identity resolution. Hundreds of sessions accumulate into a graveyard of cryptic IDs. A fresh agent — or the user — cannot answer basic questions: which sessions discussed the inscription paper? Which agent was named Reed? What decisions were made about the ontology last week? The raw storage answers none of these without reading every file.

Session ingestion transforms this raw storage into a searchable engagement index. It discovers who did what, on which project, using which method, with what outcome — and writes that knowledge into a structured record that the engagement viewer, other agents, and the user can query.

#Relationship to Architecture 39

Architecture 39 (Engagement State) defines the third externalization — persistent records of where method-tokens stand in their phase graphs. Engagement state files track engagements prospectively: an agent creates engagement.md when work begins, advances it at phase transitions, and archives it on completion. The ingestion pipeline is the archaeological complement. It reconstructs engagement history from sessions that predate the engagement state convention, or from sessions where agents did not write engagement state files. Once engagement state is standard practice, new engagements are tracked as they happen. The ingestion pipeline handles discovery after the fact.

The two are complementary, not competing. Prospective tracking (Architecture 39) captures what the agent knows at the time — phase position, obligations, participants. Retrospective ingestion captures what an external observer can reconstruct — topic, method, outcome, turn range. Prospective is richer but requires agent discipline. Retrospective is universal but approximate.

#Data Flow

Claude Code JSONL storage (~/.claude/projects/<hash>/<uuid>.jsonl)
  → Phase 1: Skeleton extraction (user messages + assistant text, metadata)
  → Phase 2: LLM engagement extraction (Sonnet, structured prompt)
  → Phase 3: Index generation (engagements.jsonl + engagements-index.md)
  → Engagement viewer (port 18820, real-time reads from index)

Phase 1 is mechanical: stream the JSONL, extract user messages and assistant text blocks, collect metadata (session ID, slug, git branch, model, timestamps, subagent count). The skeleton is a plain-text conversation transcript — role-labeled, turn-numbered, stripped of tool calls and thinking blocks. This phase handles files up to 70MB by streaming line by line.

Phase 2 is inferential: the skeleton is sent to Sonnet with a structured extraction prompt. The LLM identifies distinct engagements within the session — topic shifts, project switches, method changes — and returns structured JSON. This is the step that transforms a monolithic session into segmented, labeled units of work.

Phase 3 is indexing: each session's extracted engagements are appended to data/sessions/engagements.jsonl (one JSON record per session, containing all its engagements). A markdown index (data/sessions/engagements-index.md) is rebuilt from the JSONL, grouped by date, providing progressive disclosure: date → session → engagements.

#What Claude Code Stores

Archaeology of the JSONL format revealed what the platform records and what it omits.

Stored per session:

Full transcript: user messages, assistant responses (text, tooluse, toolresult blocks), thinking signatures
Tool invocations: every Bash command, file read/write, search — with full input and output
Timestamps: per-record, enabling duration calculation
Git branch: the active branch at session start
Slug: the session's filesystem slug
Subagent transcripts: companion directory (<uuid>/) containing subagent JSONL files
Resume traces: parentSessionId field linking resumed sessions to their predecessors
Turn durations: system records with per-turn timing

Not stored:

Cross-session lineage beyond resume: no record of which sessions are conceptually related unless one explicitly resumes another
CLAUDE.md content or hash: the bootstrap instructions that shaped the session are not captured in the session file
Model identity at top level: the model appears only inside assistant message records, requiring parsing to discover
Context budget: no record of context window size, tokens consumed, or remaining capacity
Agent names or roles: the JSONL has no field for "this agent was called Reed" or "this agent was the architect" — names appear only in transcript content (/rename commands, board post signatures)
Thinking block content: for some model configurations, thinking blocks contain only the signature ([thinking]) with content redacted

#Engagement Extraction

The core design decision: treating each session not as a single unit of work but as containing N engagements. A session with the architect might begin with reviewing a PR, shift to designing a new mandate, then pivot to debugging a viewer issue. These are three distinct engagements sharing a session container.

The extraction prompt asks the LLM to identify each engagement and classify it along seven dimensions:

Field	Description
`topic`	5–15 word description of the work
`project`	Project path if identifiable, or "system"/"meta"
`method`	One of: review, write, design, build, reorg, triage, discuss, debug, deploy, research, plan
`phase`	One of: started, in-progress, blocked, completed, abandoned, handed-off
`key_decisions`	1–3 decisions made during this engagement
`outcome`	One sentence on what was accomplished
`turn_range`	Approximate start and end turn numbers

The method vocabulary is deliberately coarse. Finer-grained method types exist in patterns/methods/ but would require the LLM to know the full method graph. The ten terms above are sufficient for discovery and filtering without domain-specific training.

Session-level metadata is also extracted: agentname, agentrole, and session_summary. These answer the "who was this?" question that the raw JSONL cannot.

#Ghost Detection

Sessions with fewer than three user messages are classified as ghosts — aborted startups, accidental invocations, sessions that never reached substantive work. Ghosts are recorded in the index (for completeness) but skip LLM extraction (for cost). The ghost threshold is a heuristic: three turns is the minimum for a meaningful exchange (greeting, request, response).

#Truncation

Skeletons exceeding 100,000 characters are truncated: the first 50,000 and last 10,000 characters are preserved, with the middle replaced by a truncation marker. This keeps the opening context (session setup, agent identity, initial direction) and closing context (final decisions, wrap-up) while dropping the mechanical middle that is typically tool output and repetitive iteration.

#Agent Identity Resolution

Agent names do not exist in the JSONL metadata. They must be discovered from transcript content through four sources, checked in priority order:

/rename commands — the user typing /rename Reed in the transcript. The ingestion script scans user messages for this pattern and captures the name.
custom-title JSONL records — some sessions contain a record type that stores user-assigned titles.
Board post signatures — agents sign board entries with [claude:opus::AgentName]. If the session's transcript contains a board post, the signature reveals the name.
Agent roster cross-reference — data/agent-roster.jsonl maps session IDs to agent names and roles. Prefix matching (the roster stores 8-character session ID prefixes) connects roster entries to full session UUIDs.

The first source found wins. If none yields a name, the session is recorded with agent_name: null.

#The Engagement Viewer

The viewer at http://127.0.0.1:18820 is the consumption layer for ingested data. It reads engagements.jsonl and the agent roster, merges them, and serves a timeline interface.

Features:

Timeline of engagements, color-coded by method (review=blue, build=green, design=purple, etc.)
Grouping by project, method, agent, or time period
Transcript panel with block-type filters — toggle visibility of user messages, assistant text, thinking blocks, tool calls, and tool results independently
Sessions / Engagements toggle — switch between viewing whole sessions and individual engagements
Search across session summaries, engagement topics, and agent names

The viewer reads from the JSONL on every request (no database, no cache beyond a 30-second directory metadata cache). This means re-running the ingestion pipeline immediately updates the viewer without restart.

#Running the Pipeline

# Full ingest — all sessions
python3 infra/scripts/session_ingest.py

# Recent only — sessions modified since a date
python3 infra/scripts/session_ingest.py --since 2026-03-15

# Incremental — skip already-ingested sessions
python3 infra/scripts/session_ingest.py --skip-existing

# Single session — by UUID
python3 infra/scripts/session_ingest.py --session <uuid>

# Dry run — extract skeletons, report sizes, don't call API
python3 infra/scripts/session_ingest.py --dry-run

The pipeline requires an Anthropic API key (from ANTHROPICAPIKEY or ~/.secrets). Each non-ghost session costs one Sonnet API call. A full ingest of 563 sessions with ~400 non-ghost sessions costs approximately $2–4 in API calls.

Output files:

data/sessions/engagements.jsonl — one JSON record per session, containing all extracted engagements
data/sessions/engagements-index.md — human-readable index grouped by date

#What This Replaces

The earlier sessionscribe.py performs a different function: it splits JSONL into round-indexed block stores (data/sessions/<agentid>/<uuid>/rounds/NNN.json) with a per-session SQLite database. The scribe is a mechanical decomposition — it makes individual blocks addressable but performs no topic discovery, no engagement segmentation, and no agent identity resolution.

The ingestion pipeline supersedes the scribe for all discovery and navigation purposes. The scribe remains necessary for block-level storage that the viewer reads when displaying full transcripts. The relationship is: the ingestion pipeline answers "what happened in this session?" while the scribe answers "show me turn 47 of this session."

#Searching the Index

infra/scripts/engagement_search.py is a fast CLI for querying the engagement index without opening the viewer.

# Full-text search across topics, outcomes, decisions, summaries, agent names
python3 infra/scripts/engagement_search.py "ontology"

# Filter by method
python3 infra/scripts/engagement_search.py --method design

# Filter by project (substring match)
python3 infra/scripts/engagement_search.py --project inscription

# Filter by agent name, role, or session slug
python3 infra/scripts/engagement_search.py --agent Reed

# Date filters
python3 infra/scripts/engagement_search.py --date 2026-03-20
python3 infra/scripts/engagement_search.py --today
python3 infra/scripts/engagement_search.py --week

# Combine filters
python3 infra/scripts/engagement_search.py "viewer" --method build --week

# Summary statistics: counts by method, project, agent, phase, day
python3 infra/scripts/engagement_search.py --stats

# Include ghost sessions (normally excluded)
python3 infra/scripts/engagement_search.py --ghosts --today

All flags:

Flag	Short	Description
`query`	(positional)	Full-text search across topic, project, method, outcome, phase, summary, agent, slug, key decisions
`--method`	`-m`	Exact match on method (review, design, build, debug, write, research, triage, reorg, plan, discuss, deploy)
`--project`	`-p`	Substring match on project path
`--agent`	`-a`	Substring match on agent name, role, or session slug
`--date`	`-d`	Exact date filter (YYYY-MM-DD)
`--today`	Today's engagements only
`--week`	`-w`	Current week's engagements only
`--stats`	`-s`	Show summary statistics instead of search results
`--ghosts`	Include ghost sessions (< 3 user turns, normally excluded)

Results are sorted by date descending and color-coded by method and phase.

#Future Work

Five extensions are planned but not yet built:

Prospective engagement tracking — Architecture 39's engagement.md files, maintained by agents during sessions rather than reconstructed afterward.
Session genealogy — reconstructing the directed acyclic graph of session relationships from temporal overlap, shared git branches, content similarity, and explicit resume links. This would answer "which sessions are continuations of which?"
CLAUDE.md version tracking — cross-referencing session timestamps against git history for CLAUDE.md to determine which bootstrap instructions were active during each session.
Context budget reconstruction — estimating context window usage from message sizes and model token limits, enabling post-hoc analysis of when agents were operating near capacity.
Automated re-ingestion — a launchd trigger that runs incremental ingestion when a session file is modified, keeping the engagement index current without manual invocation.

method . 56 . session ingestion . 2026-03-20 . zach + claude

Methods 56 — Session Ingestion — 2026 — Zachary F. Mainen / HAAK