A user discusses release work, naming, bugs, deployment, and side questions in one agent session. Hours later they ask, “Can we ship now?” Recent context may be exactly the wrong context. A good product should not make the user rescue the system by manually switching sessions.
One default session may contain release work, naming, bugs, reading, and side questions. The system decides which workstream the new input belongs to.
That state should include active tasks, facts, decisions, open loops, stale claims, evidence IDs, and the next prefix sent to the model.
Evaluation checks task success, stale-claim rejection, evidence grounding, and token usage against recent-window, summary, and simple-RAG baselines.
Plain version
Many AI products still treat a session as a chat transcript. Once a user mixes code, product planning, deployment, reading, and small side questions in one window, the system starts guessing from recent context. The user ends up manually switching sessions, copying background, or reminding the agent which topic is active.
A better product should not push that burden to the user. The user keeps talking. The system decides which topic the new input belongs to, which task it updates, which constraints still matter, which conclusions are stale, which prefix should be assembled, and how many tokens it spends.
State here does not mean a narrative summary. It should be a machine-readable object: active tasks, facts, inferences, decisions, open loops, stale claims, evidence IDs, and the assembled prefix for the next model call. Without that object, an “infinite session” is just a larger transcript.
What the challenge tests
It does not test who can call an LLM API or build another RAG wrapper. It tests a product-core capability: the append-only log is only raw material; the system must reconstruct the current workspace from it.
One session contains multiple humans, agents, startup noise, tool outputs, and topic switches.
Detect topics, tasks, facts, inferences, decisions, open loops, artifacts, and stale claims.
For the current input, choose the minimum useful context under a token budget instead of taking the latest N turns.
Evaluate the result through a single-agent input. Multi-agent fanout is allowed, but not required.
Problem statement
Build a session management layer for long-running AI workspaces. It receives chaotic append-only session logs, segments topics and tasks, builds structured session state, and assembles the minimum useful context prefix for a single agent to answer correctly or continue the task.
The history may contain multi-human and multi-agent interactions. Evaluation uses a single-agent input. Multi-agent fanout is allowed as an implementation choice, but not required. A state-centric Web demo is required because users will only trust this capability if the state is visible.
What the benchmark looks like
The benchmark does not need internal data. Organizers can provide synthetic cases. Each case is a messy session containing meaningless startup messages, multiple topics, several humans, several agents, tool results, stale conclusions, and unfinished tasks.
{"id":"e001","type":"startup","speaker":"system","text":"Agent booted. Loading tools...","topic":null}{"id":"e014","type":"user","speaker":"human_a","text":"For the release, do not rely on external API calls during startup.","topic":"release"}{"id":"e026","type":"agent_note","speaker":"agent_research","text":"The failure is probably fixed by commit 71ab.","topic":"release","status":"inference"}{"id":"e031","type":"tool_result","speaker":"ci","text":"Regression test still fails when external API is disabled.","topic":"release","evidence_id":"ci-031"}{"id":"e044","type":"user","speaker":"human_b","text":"Switching topic: keep both 'Atlas' and 'Workbench' as product name candidates.","topic":"naming"}{"id":"e057","type":"summary","speaker":"agent_writer","text":"Release is likely ready; product naming is done.","topic":"mixed","status":"stale"}{"id":"e063","type":"user","speaker":"human_a","text":"Can we ship now? Also what remains open on naming?","topic":null}What teams build
Required deliverables
- A runnable Web demo that exposes state, not just chat.
- `session_state.json`: topics, tasks, facts, inferences, decisions, open loops, artifacts, and stale claims.
- `answers.jsonl`: answers to benchmark questions with evidence IDs.
- `token_usage.json`: token usage for prefix assembly and final answers.
- `implementation_review.md`: state model, prefix policy, model choices, and trade-offs.
Bonus directions
- Topic graph showing dependencies, conflicts, and returns.
- Prefix diff comparing append-only baseline and assembled prefix.
- Stale invalidation that marks which old claims are no longer valid.
- Multi-agent fanout for state construction or verification.
- LLM judge for implementation review, without replacing deterministic checks.
Why the Web UI matters
This cannot be a CLI-only challenge. A CLI can prove the algorithm runs, but not the product experience. The product promise is: you no longer need to switch sessions. To trust that promise, users must see the system’s topics, active tasks, open loops, evidence, and token usage.
That is why the banner, cover, information hierarchy, and state panels matter, but not as decoration. Session state is invisible by default. Good UI turns working memory into something users can understand, inspect, and correct.
- Topic list: which topics exist in the session and which remain active.
- Active task view: which task the new input hits and which old constraints it depends on.
- Stale-claim warnings: which old summaries or agent notes should no longer be trusted.
- Evidence links: key claims jump back to concrete event or resource IDs.
- Prefix preview or diff: show what is actually sent to the model and how it differs from an append-only baseline.
- Token budget: show how many tokens the prefix and final answer spend.
How it is evaluated
| Dimension | Weight | What to evaluate |
|---|---|---|
| Task performance | 35% | Whether answers are correct and recover constraints, open loops, and evidence across topics. |
| Evidence grounding | 15% | Whether key claims cite the correct event/resource IDs and distinguish facts from inferences. |
| Token usage | 20% | Whether it beats recent-window, narrative-summary, and simple-RAG baselines with less context. |
| State design | 15% | Whether session state is clear, machine-readable, and reusable by later agents. |
| Web product quality | 10% | Whether state is visible, understandable, inspectable, and usable as a product. |
| Implementation review | 5% | Human and LLM-assisted review of schema, routing, prefix policy, trade-offs, and reproducibility. |
Technical boundaries
- The stack is open: open-source or commercial LLMs, vector DBs, graph DBs, context DBs, agent harnesses, and browser automation are all allowed.
- “We used RAG” is not enough. Teams must explain the session state schema, routing policy, prefix assembly, and token accounting.
- Final solutions should be open source. The benchmark evaluates not only the result, but also whether the implementation is reproducible and worth learning from.
- Multi-agent systems are allowed but not required. A strong single-agent implementation with a good session manager can score highly.
Three-day plan
| Time | What to finish |
|---|---|
| Day 1 | Understand the benchmark, parse session logs, define the state schema, and run an append-only baseline. |
| Day 2 | Implement topic/task routing, state extraction, prefix assembly, token accounting, and start the Web UI. |
| Day 3 | Run all cases, improve task quality and token usage, finalize the state UI, report, and implementation explanation. |
Compressed version
The goal is not to make sessions literally infinite. The goal is to hide session boundaries from the user by translating an infinite-session experience into correct working state under finite token budgets.