Nemotron 3.5 Lightning Turns Agent Speed Into a Routing Problem
NVIDIA's newest open agent model loses the obvious benchmark contest. In the company's own consistent-harness comparison, Qwen 3.6 35B A3B beats the BF16 version of Nemotron 3.5 Lightning on 13 of 14 capability rows.
That sounds like a poor opening for a model launch. It is also the fact that best explains why this release matters.
Nemotron 3.5 Lightning is a 30-billion-parameter, text-only model that activates roughly 3 billion parameters per token. NVIDIA did not build it to be the smartest model in an agent stack. It built Lightning to be the model that handles the repetitive turns: inspect a tool result, make a bounded edit, format an answer, run the next command, check a postcondition, and escalate when the task stops being routine.
The obvious take is “a fast open MoE model for agents.” The more useful take is that NVIDIA has packaged an execution lane: weights, quantization, reasoning controls, tool parsers, evaluation recipes, three speculative-decoding options, local runtimes, and an experimental router. The model's value is inseparable from that stack.
For builders, this changes the evaluation question. Do not ask whether Lightning can replace a frontier planner. Ask whether a precisely configured Lightning worker can close enough validated turns per dollar to justify a place underneath one.
Start with what NVIDIA actually measured. The BF16 model card puts Lightning beside Qwen 3.6 35B A3B under a common evaluation harness. Qwen leads on 13 of the 14 listed rows. Lightning's only win is IFBench loose, at 71.88 versus 63.71.
Lightning reports 51.56 on SWE-bench Verified, 24.58 on Terminal-Bench 2.1, 36.97 on BrowseComp, and 9.28 on tau3-bench Banking. These are NVIDIA-run results, and the company's reproducibility notes warn that runtime versions, parsers, cache types, topology, and sampling can move scores. They are not universal rankings. They are still enough to reject the idea that “fast” means “safe to use as the default autonomous brain.”
The right division of labor is therefore narrow by design. Let Lightning do work with deterministic postconditions: the test passes, the schema validates, the expected record exists, the diff stays inside scope. Send ambiguous planning, novel debugging, policy-sensitive decisions, and stubborn failures to a stronger model or a human.
The architecture is unusual enough to matter operationally. The published NVFP4 configuration contains 52 layers: 23 Mamba-2 layers, 23 sparse mixture-of-experts layers, and six attention layers. It defines 128 routed experts plus one shared expert, routes each token to six experts, and includes one multi-token-prediction layer.
“3B active” describes compute, not storage. The official Hugging Face shards total about 65.83 GB for BF16 and 21.56 GB for NVFP4. Add caches, recurrent state, runtime overhead, and a speculative draft model before deciding what really fits. A 3B-active model is not a 3B-weight model wearing a clever label.
NVIDIA also ships reasoning on/off controls, a runtime reasoning budget, OpenAI-compatible serving recipes, and explicit tool/reasoning parsers. The vLLM path names a Qwen3 Coder tool parser and a Nemotron v3 reasoning parser. Launch-day recipes include vLLM 0.27.1 or nightly images, a TensorRT-LLM release candidate, and an SGLang development image.
That makes the deployable unit this tuple:
checkpoint × quantization × runtime × parser × cache policy × max context × decoder × concurrency × validator × route
This is the same operational shift I wrote about in vLLM 0.27.0's expanding runtime surface. “Supported” is no longer a Boolean. It is a tested combination of versions and settings. Change one element and you may have changed latency, memory use, tool syntax, reasoning length, or task accuracy.
A production Lightning lane needs two decisions: which model should receive the turn, and how that model should decode under the current load.
NVIDIA's NeMo Switchyard supplies the first router. It can translate among OpenAI Chat, OpenAI Responses, and Anthropic Messages, then select backends with classifier, stage, escalation, or random-routing strategies. Its README is equally clear about maturity: Switchyard is pre-alpha, its APIs may change, and it is not for production use.
The second router is easy to miss because it sits below the model name. Lightning ships with native multi-token prediction plus DSpark and DFlash draft models. NVIDIA does not recommend one universal winner. The NVFP4 deployment card recommends DSpark for DGX Spark and some low-concurrency data-center work, while an H100 maximum-throughput recipe runs without speculative decoding. The useful choice changes with queue depth, concurrency, hardware, prompt shape, and latency target.
That means a static image with “speculation enabled” leaves performance on the table. Semantic routing and decoding-policy routing are different feedback loops:
Use for bounded edits, extraction, formatting, tool-result follow-up, and actions with deterministic postconditions. Track accepted outcomes and escalation rate.
Use for ambiguity, novel failures, cross-domain planning, and tasks where the cost of a plausible wrong answer exceeds the savings from cheap turns.
Evaluate DSpark, DFlash, or MTP when per-user latency matters and draft acceptance is high enough to repay verification overhead.
At high concurrency, no speculative decoder may win. Let queue depth and measured throughput choose, not launch-day defaults.
My bet is that agent gateways will eventually expose both dimensions as first-class policy: capability tier and decode profile. A single model ID is too coarse for an always-on system.
The NVFP4 configuration declares a 1,048,576-token maximum. That is useful, but the headline hides three qualifications.
First, a context ceiling is not evidence of reliable reasoning across every position in a million-token trace. Second, NVIDIA's single-H100 BF16 guidance uses 256K; its 1M BF16 recipe requires an override on an eight-H100 or GB200 setup. Third, long histories consume the cache and Mamba-state headroom that could otherwise serve more concurrent turns.
This creates a direct conflict. Lightning is positioned as a high-volume worker, yet casually admitting giant histories reduces the volume it can handle. The production answer is not to give every agent a million-token diary. It is to create a separately admitted long-context tier, then use retrieval, compaction, checkpoints, and bounded histories for routine workers.
That complements, rather than replaces, the session and memory policies discussed in SGLang 0.5.17. Context length is a capacity reservation. Treat it like one.
It is tempting to treat NVFP4 as the same model in a smaller box. NVIDIA's own table says otherwise. Quantization changes the capability shape unevenly: some scores fall, some rise, and the largest regressions are not necessarily in the lane your generic smoke test covers.
| Evaluation | BF16 | NVFP4 | Operator read |
|---|---|---|---|
| AA-LCR | 52.00 | 49.19 | Long-context regression deserves lane-specific testing. |
| SWE-bench Multilingual | 39.33 | 36.47 | Do not infer multilingual coding parity from English checks. |
| Humanity's Last Exam | 11.72 | 10.47 | Hard-reasoning capability moves down. |
| SWE-bench Verified | 51.56 | 52.80 | A small gain shows quantization effects are not monotonic. |
| IFBench loose | 71.88 | 72.88 | Basic instruction following holds up well. |
These remain vendor-run measurements, not a universal quantization law. The engineering lesson is narrower and stronger: a route should name the exact checkpoint and serving stack. Certify BF16 and NVFP4 separately on the tasks each will actually receive.
NVIDIA says Lightning reaches up to four times the output speed of similar-sized models. Its public chart places the NVFP4 checkpoint near 667 output tokens per second, but the footnote says that point came from pre-release testing on a private DeepInfra deployment. At launch, Artificial Analysis had independently run capability tests but no public Lightning provider-speed result. The headline speed has not yet been independently reproduced on a public endpoint.
The better vendor claim is outcome-shaped: NVIDIA says Lightning used roughly 30% fewer H100 GPU-hours than Qwen 3.6 35B to complete 10,000 PinchBench tasks at similar accuracy, around 86%. That includes more of the actual job than a single token-rate point, though it is still NVIDIA's result.
Launch partner testing adds useful detail. Thoughtworks reports native MTP delivered 1.46–1.96 times unaccelerated throughput across 2,091 H200 and B200 measurements. Its worked cost model moved from $0.477 to $0.250 per million output tokens. Those are direct early-access tests, but Thoughtworks participated in the launch and the dollar result depends on its hardware assumptions.
Then comes the twist: raw token speed may flatter a verbose model. Artificial Analysis recorded about 104.1 million output tokens for its Lightning index run, roughly 30,900 per task and well above its comparable-model median. If extra reasoning, retries, or validation consume the speed dividend, tokens per second becomes the wrong denominator.
Measure this instead:
accepted outcomes ÷ (wall-clock time × total system cost)
Total system cost includes failed attempts, verifier calls, frontier escalations, idle GPUs, power, orchestration, and human cleanup. “Always on” only creates good self-host economics when enough work arrives to keep the hardware busy.
LangChain's controlled Switchyard test used 145 Deep Agents tasks. Claude Opus 4.8 alone scored 86.0% at $11.45 per run. Lightning alone scored 77.7% at $0.72. A routed arm scored 80.0% at $3.00, sent 93% of calls to Lightning, and cut cost 74% versus Opus alone.
The six-point accuracy loss versus Opus is real. LangChain also says the routed arm's 2.3-point gain over Lightning alone was within run variance. This is not proof that routing magically preserves frontier quality. It is proof that builders can put a price on the quality they surrender.
That trade can be rational in a lane where validators catch failures and humans do not have to clean up silent mistakes. It can be reckless in banking, production changes, or external actions where a six-point loss has asymmetric consequences.
NVIDIA's openness claim is meaningful, but it needs precise language. The OpenMDW 1.1 license grants broad rights to use, modify, and distribute model materials and does not restrict generated outputs, subject to notices and other license terms.
The data story is less frictionless. The model card says only a sample of released training data is ungated; remaining code, math, and multilingual data requires approval, and the inventory includes private NVIDIA and third-party sources. “Weights, data, and recipes” describes a strong reproducibility direction, not downloadable access to every training input.
The released Agentic Terminal Pivot dataset is still unusually useful. It contains 31,111 samples drawn from 2,716 successful trajectories across 630 vendor-authored terminal tasks. GLM-5.1 drove the teacher agent, and only verifier-passing trajectories were retained. That provenance is both a strength and a caution: the data has executable success signals, but it is narrower and cleaner than a messy production terminal.
Once the starting stack is reproducible, durable advantage shifts away from simply possessing the base weights. The moat becomes your private failure set, deterministic validators, route telemetry, escalation policy, and adapter cadence. Thoughtworks' launch tests point in that direction: a legal LoRA moved CaseHOLD accuracy from 35% to 77%, while a healthcare adapter produced smaller gains. Specialization works best when the lane and the failure evidence are specific.
Everything in this section is my interpretation of the sourced evidence above.
First, agent platforms need two-dimensional routing. Model choice answers “how much capability?” Decode choice answers “how should this checkpoint run under the current queue?” Those controls will converge in gateways, but operators should measure them separately.
Second, the million-token tier will become a scarce pool. Long histories and cheap parallel execution pull in opposite directions. Serious deployments will admit giant contexts explicitly, then keep ordinary workers on summarized state and retrieval.
Third, the exact deployment tuple becomes an auditable artifact. A route to “Nemotron Lightning” is underspecified. A production record should identify checkpoint hash, quantization, runtime image, parser versions, cache dtype, context cap, decoder policy, and evaluator version. That is the evidence needed to reproduce an incident.
Fourth, public speed gaps will narrow after normalization. Private pre-release endpoints and carefully chosen concurrency profiles usually flatter launch numbers. I still expect Lightning to remain unusually competitive per active parameter because the architecture, three draft paths, and partner MTP results provide a credible efficiency base.
Fifth, fine-tuned workers will matter more than the base model. A cheap, permissively licensed execution model becomes strategic when it learns one company's procedures and failure modes. The planner may remain a rented frontier API. The worker can become local, specialized, and replaceable.
That last split matters. Durable orchestration state should not live inside the worker model. The runtime migration issues in the OpenAI Agents SDK release reinforce the point: replay, approvals, protocol negotiation, and checkpoints belong in the agent system. Lightning is one execution engine beneath it.
Do not begin with a blended benchmark called “agent quality.” Define lanes first: bounded code edits, retrieval follow-ups, structured extraction, terminal actions with postconditions, tool-result validation, or customer-facing decisions. Each lane should have its own cost of error and escalation rule.
For a low-friction local trial, Ollama 0.32.9 added the Nemotron 3 architecture, and its library lists a roughly 25 GB package with a 1M ceiling. The MLX variant is smaller and capped at 256K. That is a good way to inspect behavior, not proof that a laptop profile matches the data-center performance claim.
The first production pilot should be boring: one narrow lane, one validator, one stronger fallback, one quantization, and a replayable trace. Add complexity only after the failure data says where it pays.
Teams with high-volume, repetitive agent turns and strong validators should test it immediately. That includes code platforms with testable edits, internal automation with schema and state checks, research agents with bounded retrieval steps, and enterprises that need local weights or private adapters.
Teams looking for one model to plan, execute, judge, and authorize its own consequential actions should wait. Lightning's own benchmark profile argues against that design.
The most interesting comparison is not a frontier API. It is another local agent worker. Meta's Muse Glimmer is dense, multimodal, and capped near 131K context; Lightning is text-only, sparse, 3B-active, and designed around throughput and a 1M ceiling. Choose based on workload shape, not the shared “30B agent model” label.
No. It has roughly 30 billion total parameters and activates around 3 billion per token. Active parameters help explain compute efficiency, but the weights, caches, recurrent state, draft model, and runtime determine memory requirements. Official shards are roughly 65.83 GB for BF16 and 21.56 GB for NVFP4.
The NVFP4 configuration declares a 1,048,576-token maximum, but deployment profiles differ. NVIDIA's BF16 single-H100 guidance uses 256K, while the 1M BF16 recipe uses an override on eight H100s or GB200. Maximum context also says nothing by itself about reliable million-token reasoning or useful concurrency.
Not at launch. NVIDIA's chart used private pre-release DeepInfra testing for Lightning's speed point. Artificial Analysis independently measured capability, not a public provider-speed result. Thoughtworks' launch-partner MTP tests provide supporting throughput evidence, but they are not a reproduction of the headline comparison.
Usually no. It is better understood as a worker tier under a stronger planner, with deterministic validators and escalation. LangChain's controlled test showed major cost savings, but routed accuracy remained six points below Opus 4.8 alone.
The model materials use the permissive OpenMDW 1.1 license, and NVIDIA publishes weights, recipes, evaluation material, and useful datasets. The model card also says parts of the data require approval and lists private sources. Call it a strong open release, not zero-friction access to every training input.
Nemotron 3.5 Lightning is valuable because NVIDIA did not pretend every agent turn deserves frontier intelligence. It packages a plausible workhorse for the repetitive middle of long-running tasks, then surrounds it with the pieces needed to tune throughput, specialize behavior, reproduce evaluations, and escalate hard work.
The bargain is specific. Builders accept lower broad capability in exchange for control over weights, runtime, quantization, context, and cost. That bargain only works when the lane is bounded and the outcome can be checked.
The model launch will get attention for 3B active parameters, a million-token ceiling, and a large token-per-second number. The durable idea is quieter: the execution model is becoming a replaceable component, while routing policy, validation, and failure data become the product.
That is why Lightning matters—even before it wins a benchmark row.