The Allen Institute for AI (Ai2) introduced StateBench on November 3, 2026, a research benchmark that separates finding information in a long document from following changes through it. For workers asking AI to make sense of project histories or changing records, that distinction matters: retrieving the right passage may not reveal what is true now.
A model’s context window is the amount of material it can take into account in one request. That capacity alone does not establish how accurately it handles successive updates. StateBench’s research preprint, submitted on November 2, offers a controlled way to examine the difference—not a measurement of workplace error rates.
The answer may never appear in one sentence
Consider a hypothetical task record: it begins with 12 open tasks, then says three were closed and two were added. Assuming the entries are complete, in order and concern the same task pool, the current count is 12 − 3 + 2 = 11. The latest sentence alone does not give that answer. A later entry explicitly stating that 11 tasks remain would turn the question into a lookup.
The difference is between finding a recorded value and following a chain of changes.
StateBench’s task generator distinguishes four kinds of update. In plain language, they are:
Unchanged facts. A value is assigned and stays fixed; the job is to recover it.
Replacements. A later entry supplies a new value. The earlier value no longer determines the answer.
Accumulating changes. An update modifies the current value, so earlier steps still matter. The task-count example illustrates this idea; the benchmark’s arithmetic instead wraps within a fixed range.
Dependent changes. One record takes another’s current value. Correctly copying Team A’s count into Team B’s record, for example, requires knowing A’s count at that moment. A later change to A does not alter the number already copied.
The researchers construct matched cases with the same number of variables, distinct values and positions of assignments, while changing the operations required. That makes the comparison more informative than simply giving one task a longer document: it helps isolate the work needed to reach the final answer.
What the experiments found
In Ai2’s reported evaluation of four proprietary models, the models handled unchanged values and explicit replacements well. Accuracy fell on longer sequences involving cumulative or dependent updates. On the longest cases in the hardest category, Ai2 reports accuracy of 13% or lower. The models were asked to generate the final values of all variables; this is not a percentage of wrong business decisions or a score for every assistant configuration.
Presentation also mattered in experiments with smaller open models: equivalent problems could be much easier in one familiar benchmark format than when presented as spreadsheet edits. For a buyer, that supports asking for evidence in the format their team uses—not assuming a successful demonstration transfers unchanged.
The paper’s scope is limited to a fixed set of variables. It does not cover freely adding new people or objects throughout a conversation, and leaves the broader prevalence of hard state tracking in natural data unresolved. These synthetic results cannot tell a project manager how often an assistant will misunderstand their records, or establish a universally safe document length.
The code and dataset are publicly accessible. The dataset includes update sequences and expected final states, making the tasks inspectable. These are research materials, not a ready-made assistant for workplace documents.
Ask for evidence that matches the records
The practical implication is to assess input capacity and reconstruction accuracy separately. For a team evaluating an assistant, three checks follow from that distinction:
Use histories with independently known answers. Include the replacements, running adjustments and dependencies that matter in your records, in their actual format. Establish the correct final result without relying on the assistant being evaluated. Missing entries, conflicting updates and newly introduced entities need separate coverage.
Define what counts as correct. Distinguish getting every required field right from a partial-field average or choosing a supplied answer. Keep the model version, prompt and tools alongside the result. Check the ordered updates: citations can identify relevant passages without proving every dependency was handled.
Include the checking effort. Assess the work needed to verify and correct the result alongside response time and usage cost. A fast answer that needs extensive reconstruction by a reviewer may not meet the task’s needs.
For teams keeping long conversations, RohitAI’s explanation of Claude message threads addresses storage, continuation and recovery. Those are separate questions from whether the current facts have been reconstructed correctly; StateBench does not evaluate that feature.
A useful demonstration should show the correct current picture after the relevant changes—not just the passage where one fact appeared. That is the evidence to seek before treating a large context window as a reason to rely on an answer.