AstaBrief 8B: Self-Hosting Cited Scientific Reports

On October 2, 2026, Ai2 announced the open-sourcing of AstaBrief 8B, a model for teams building scientific literature assistants. AstaBrief turns a research question and retrieved excerpts into a cited report. Teams can run that writing stage on their own infrastructure; document processing, retrieval and scientific review still need to be supplied.

The practical starting point is an existing literature-search pipeline with usable passages and stable source identifiers. This guide explains how to connect the checkpoint, budget its context and review its output. It is based on published documentation, not an executed installation or model benchmark.

What you can self-host

Ai2 says AstaBrief powers Asta’s hosted Fast mode. The downloadable model offers a separate deployment route; using the hosted product is not the same as processing a private corpus locally. The ScholarQALite implementation accepts reranked evidence, generates the report body in one pass, parses citations and makes a separate title-generation call. It is not a PDF extraction or indexing engine.

The checkpoint descends from Qwen3-8B through supervised fine-tuning and direct preference optimization (DPO). The model card labels the weights Apache-2.0 and describes research and educational use. The separately released SFT and DPO dataset cards carry CC BY-NC-4.0 labels and identify third-party model-output terms. Treat checkpoint deployment and training-data reuse as separate decisions; the release is not uniformly Apache-licensed.

For confidential work, inspect the full data path. The ScholarQA README documents external retrieval, configurable model and reranker services, and optional tracing. Local report generation does not by itself make document parsing, embeddings, reranking, query preprocessing, title generation, fallback calls or logs local. Those components need their own deployment choices.

Prepare evidence with recoverable source locations

AstaBrief needs a question plus retrieved passages, not a folder of PDFs. Before generation:

• Extract text from documents you are permitted to use. Keep a stable document identifier and the original page, section or text offsets alongside each passage.

• Retrieve and rerank passages against the question. Preserve enough surrounding material to interpret methods, populations and limitations; decide explicitly which evidence fits the context budget.

• Build a reference-key-to-passage mapping and retain a separate lookup from each key to its document metadata and source location.

Ai2’s reference formatter groups snippets by paper, orders them by source position and retains provenance fields. Its parser expects citation keys shaped like [corpus_id | Author | year | Citations: N]. For private papers, adapt the mapping and parser to the metadata you actually have. Do not invent citation counts or reuse unrelated public paper IDs just to satisfy the format.

Connect the checkpoint to the supplied prompt

Select allenai/AstaBrief_8B for the final DPO checkpoint. The public, ungated repository was checked at revision 045f1dcca8f663735d4f57dcd4e92deeb2c83533. Record the model, tokenizer, prompt and application-code revisions together. The card’s inference example names allenai/AstaBrief_8B_SFT, so copying it unchanged would select a different checkpoint.

The supplied prompt has slots for [QUERY] and [SECTION_REFERENCES]. Fill them with the question and serialized reference mapping, then apply the checkpoint tokenizer’s chat template to one user message. Preserve the SECTION; and TLDR; markers if you use Ai2’s parser.

For a vLLM integration, load an AutoTokenizer and one LLM engine from the same pinned checkpoint; do not also load an unused Transformers model. The vLLM quickstart documents llm.generate and says it does not apply the chat template automatically. The fragment below assumes those instances and the prepared query, references and template already exist. It illustrates the interface, not a tested runtime configuration.

import json
import re
from vllm import SamplingParams

# tokenizer and llm use the same pinned final checkpoint.
# template is Ai2's saved sft_prompt.txt; references maps keys to passages.
values = {
    "[QUERY]": query,
    "[SECTION_REFERENCES]": json.dumps(references, ensure_ascii=False),
}
prompt = re.sub(
    r"\[QUERY\]|\[SECTION_REFERENCES\]",
    lambda match: values[match.group(0)], template,
)
formatted = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=False,
    add_generation_prompt=True,
)
input_tokens = len(tokenizer.encode(formatted, add_special_tokens=False))
if input_tokens + 4096 > 16000:
    raise ValueError("Reduce the evidence to the planned context budget")

params = SamplingParams(
    temperature=0.7, top_p=0.95, max_tokens=4096,
    stop_token_ids=[tokenizer.eos_token_id],
)
completion = llm.generate([formatted], params)[0].outputs[0]
if completion.finish_reason == "length":
    raise ValueError("Report hit the output limit; review before displaying")
raw_report = completion.text

The sampling values follow the card’s illustration; they are not an optimized setting for every corpus. Choose and validate a runtime build for your hardware separately. The 16,000-token check is a pilot budget explained below, not a newly discovered model limit.

There is also a policy choice in the template: it permits content from the model’s own knowledge labeled LLM Memory, requests uncited section summaries and states that the current year is 2025. An evidence-only application must decide how to handle each of those behaviors. Changing the prompt or date creates a variant to evaluate; it is not a proven drop-in improvement. Read the exact template before adopting it.

Budget context and memory separately

Ai2 reports DPO training with a maximum sequence length of 16,000 tokens; the checkpoint configuration declares 40,960 positions. The latter is not evidence that scientific synthesis remains reliable at that length. For a first pilot, reserving 4,096 output tokens within 16,000 leaves at most 11,904 tokens for the question, evidence and template. Count the fully formatted input, not just the excerpts.

A memory estimate also needs more than the “8B” label. The pinned weight index lists 16,381,470,720 tensor bytes, or about 15.26 GiB. Using the configuration’s 36 layers, 8 key/value heads and head dimension of 128 gives the following illustrative cache calculation:

BF16 KV cache per token
= 36 layers × 8 KV heads × 128 dimensions × 2 (K and V) × 2 bytes
= 147,456 bytes

Calculation assumptions: one sequence, conventional full-attention caching, BF16 keys and values, and no cache compression, offload or prefix sharing; 1 GiB = 2³⁰ bytes. Totals exclude activations, runtime buffers, allocator overhead and other pipeline components. This is not measured VRAM or a guarantee of fit on a particular GPU. Concurrent unshared sequences add cache demand; quantization changes the assumptions and needs its own quality checks.

Review the claim, not just the citation link

The released response parser normalizes references and connects identifiers or author mentions to metadata. It also removes citation and memory labels from TLDR text. These are formatting operations, not checks that a passage supports a scientific statement. Keep the raw output alongside the rendered report so review does not lose those distinctions.

Before a report is shown as reviewed, check four things:

• Identity: each citation resolves to the supplied evidence map, with its supporting passage and original location available.

• Support: the passage actually backs the attached statement; a related paper title is not enough.

• Scope: the statement preserves the study’s population, conditions, time frame and uncertainty instead of broadening its conclusion.

• Coverage: summaries and uncited claims receive review too. Flag unknown references, incomplete sections and evidence gaps rather than silently presenting them as settled answers.

The scope check follows a limitation Ai2 explicitly discusses: citation support does not capture every way a report can overgeneralize. The same distinction between a recorded result and its scientific validity appears in RohitAI’s BootLoops guide, which concerns computational tools rather than literature synthesis.

Read the benchmarks as historical pipeline evidence

Ai2 reports mean full-pipeline times of 51.1 seconds for Fast mode and 178.5 seconds for Thinking mode. Dividing those means gives a calculated 3.49× ratio. The faster path also removes summarization, clustering and section-by-section writing stages, so the result does not isolate model token speed or establish local latency or dollar savings. Most training and evaluation occurred in 2025; Ai2 says it has not rerun the full comparison against current frontier models. Source: Ai2’s announcement.

The model card’s reported scores also vary by benchmark:

Higher is better within each column, but the benchmarks use different metrics; scores are not comparable across columns. AstaBrief does not lead either test column. Ai2 separately reports a 72% LLM-judged pairwise win rate against Asta ScholarQA on the CS2 test questions. That is neither human preference nor factual accuracy, and AstaBrief was optimized for pairwise report ranking during DPO. These results justify a corpus-specific evaluation, not a universal replacement claim. Ai2’s evaluation notes explain the distinction.

A useful first pilot

Keep the incumbent generator and AstaBrief on the same frozen corpus, retrieved passages and held-out questions. Include conflicting findings, narrowly scoped studies and questions the corpus cannot answer. Have domain reviewers judge relevance, completeness, citation support and claim scope separately, ideally without seeing system identities. This is a proposed evaluation, not one performed for this article.

Record the model and prompt identities, runtime and hardware, inputs and outputs, UTC timestamps, sampling settings, token lengths, peak memory, phase latency and end-to-end latency. Retain the reviewer method and limitations with those records so any later performance claim can be checked independently.

No verified per-report cost or self-hosting break-even point was found in the inspected release material. Budget compute utilization, document processing, storage, retrieval and expert review separately. AstaBrief is a candidate when a team already controls the evidence pipeline and wants to host report generation; it is not a shortcut around building that pipeline or deciding when its answers are trustworthy.

Methodology: AI-assisted reporting and analysis based on Ai2’s announcement, model and dataset cards, pinned configuration and source code, and vLLM documentation, checked October 2, 2026. Performance figures are Ai2-reported; context and memory estimates are explicit calculations. No model inference, PDF workflow, privacy audit or hardware benchmark was performed.