Inkling-Small Puts Open Weights Back in the Agent Router
Thinking Machines has released Inkling-Small, and the word Small is doing dangerous amounts of work.
The model activates 12 billion parameters per token, which makes its inference profile much lighter than the 41B-active Inkling. But it still carries 276 billion total parameters. Thinking Machines lists a 180GB aggregate VRAM minimum for the NVFP4 checkpoint and 600GB for BF16. This is not a model you casually drag onto a developer laptop.
That does not make the release less important. It makes the deployment thesis more specific.
Inkling-Small is a serious attempt to put an open-weight model back into the high-volume layer of an agent stack: coding, tool calls, repository transforms, grading, synthetic data, retrieval-backed internal work, and other procedural jobs that do not deserve a closed frontier model on every turn. Its continuous reasoning-effort control gives a router another knob. Its native image and audio inputs widen the work it can accept. Its Apache 2.0-labeled weights, NVFP4 checkpoint, LoRA support, and launch-day serving integrations make it unusually deployable for a model of this scale.
The obvious take is that Thinking Machines compressed Inkling into a cheaper sibling. The evidence points somewhere more useful. Inkling-Small nearly matches Inkling on a composite intelligence index, beats it on several reasoning and coding tests, and loses badly on factual knowledge. That is not a smaller copy. It is a different capability shape.
The teams that benefit will be the ones that route around that shape.
This is also the promised follow-up to RohitAI's first look at the Inkling family. On July 15, Inkling-Small was still a preview and arguably the more strategically interesting checkpoint. The final weights, hardware requirements, API prices, and independent results now let us test that argument.
An agent router usually starts with a crude ladder: cheap model for easy work, expensive model for hard work. Inkling-Small suggests a better design. Route on the kind of uncertainty, not merely a guessed difficulty score.
A practical router changes effort and grounding before it changes the model. Factual uncertainty and consequential actions still trigger escalation.
The first route is low-effort transformation: classification, extraction, formatting, and short deterministic edits. The second is higher-effort procedural work: coding, terminal use, tool recovery, and multi-step transformations. The third adds retrieval for internal knowledge. Only the fourth escalates to a stronger model when the task depends on broad recall, high-stakes judgment, or a long horizon with expensive failure.
That design uses Inkling-Small's strengths without pretending its weaknesses disappeared.
Thinking Machines says it changed the pretraining mixture and recipe, used on-policy distillation from Inkling, and then continued agentic coding reinforcement learning for two weeks after the preview checkpoint. That is a provider account, not an independently audited training ledger. Still, the before-and-after pattern is unusually revealing.
The final launch table and the July 15 preview are not a clean scientific ablation. Harnesses matter, and the company labels the 64.7% Terminal-Bench number as its best internal harness. Artificial Analysis measured roughly 55% on the same benchmark family.
But the direction is hard to miss. The largest improvement landed exactly where the continued training was aimed: procedural coding work. Factual recall did not come along for the ride.
That is the first non-obvious lesson from this release: targeted post-training can change which work belongs on a smaller model without making the model generally more knowledgeable. For builders, that can be more valuable than moving every benchmark by two points. A router needs a worker with a dependable job description, not a vague claim of greater intelligence.
Artificial Analysis gives Inkling-Small a score of 40 on its Intelligence Index, compared with 41 for Inkling. DeepSeek V4 Flash ties Small at 40 with a similar 284B-total, 13B-active architecture. If you stop there, the smaller checkpoint looks like a near-complete substitute.
It is not.
On AA-Omniscience, Inkling-Small scored -9 against Inkling's +2, with 31% accuracy versus 40%. Interestingly, Small's hallucination rate was lower, 57% versus 63%. It was not simply more reckless. It knew less often.
The distinction matters in production. A model can be careful about uncertainty and still lack the fact needed to finish a task. Retrieval may repair that. More reasoning effort may not.
| Workload | Evidence for Inkling-Small | Main failure risk | Recommended route |
|---|---|---|---|
| Repository edits and terminal tasks | Strong provider coding scores; solid independent Terminal-Bench result | Harness sensitivity and long-horizon drift | Use Small with tests, tool traces, and bounded permissions |
| Internal knowledge synthesis | Strong instruction following and tool use | Missing facts presented inside polished work | Add retrieval, citations, and abstention checks |
| Unaided factual analysis | Near-frontier composite score can look reassuring | Weak SimpleQA and AA-Omniscience results | Escalate or require search-backed evidence |
| Image and short-audio workflows | Native multimodal input and competitive provider benchmarks | Limited independent multimodal reproduction | Pilot by modality before combining them |
There is another useful warning in the independent results. Inkling-Small reached 917 Elo on AA-Briefcase versus Inkling's 839 and completed tasks in 34 turns on average instead of 81. Yet their rubric pass rates were nearly tied at 20% and 19%. Better presentation and fewer turns are valuable, but polished brevity is not the same as substantive completion.
Inkling-Small is a 42-layer sparse mixture-of-experts transformer. Each token routes to six of 256 experts, plus two shared experts. The architecture explains how 12B active parameters can deliver capability associated with a much larger stored model.
It also explains why two numbers must travel together:
12B active → approximate per-token expert compute
276B total → weights, memory, loading, interconnect, and procurement
Thinking Machines' model card lists at least 600GB of aggregate VRAM for BF16: four B300s or eight H200s. The official NVFP4 checkpoint lowers that to 180GB. W4A4 can fit on one B300 with SM100-or-newer hardware; W4A16 needs two H200s.
So “Small” moves Inkling from a large institutional deployment toward a one-server deployment on cutting-edge hardware. It does not move the model into ordinary local AI.
This is the second useful insight: active parameters price the hot path, while total parameters price the operating model. Capacity planning still has to cover KV cache, expert communication, replicas, failover, batching, and cold starts. A 180GB minimum is not a comfortable production envelope.
The runtime story is better than a typical weights drop. vLLM 0.26.0 shipped an Inkling-family stack before the final Small release, including hybrid-attention work, speculative decoding, LoRA, and ModelOpt NVFP4 support. The official repository also links SGLang, Transformers, TokenSpeed, and Unsloth paths. That is evidence of launch-day operational work, not a guarantee that every path produces identical behavior.
Record the whole deployment tuple:
checkpoint + quantization + runtime + GPU topology
+ context tier + effort + renderer + tool parser + harness
That tuple is what your eval certifies. “Inkling-Small passed” is too imprecise to be useful.
The downloadable checkpoint supports up to one million tokens. The managed products available at launch do not expose that same ceiling.
Tinker's fine-tuning tiers are 64K and 256K. Its first-party serverless inference endpoint is 256K. Artificial Analysis also evaluated a roughly 262K context route. The serverless documentation explicitly calls inference beta and says it is not recommended for intensive production use yet.
That creates three distinct products:
Do not blend them into one capability claim.
Long context also magnifies the infrastructure side of the MoE story. More tokens mean more KV-cache pressure, longer prefill, larger failure domains, and more opportunity for tool state to go stale. A million tokens can be useful for deep repositories, long audio-derived transcripts, or document estates. It can also be an expensive way to avoid building retrieval and memory well.
My expectation is that early production adopters will care more about cache hits, 256K reliability, and tool-state coherence than about reaching the checkpoint ceiling.
Inkling-Small accepts a continuous reasoning-effort value from 0.0 inclusive to below 1.0. Thinking Machines maps named presets to none at 0.0, minimal at 0.1, low at 0.2, medium at 0.7, high at 0.9, and xhigh at 0.99. Its documentation warns that higher effort does not guarantee higher accuracy, and that effort is independent from the maximum generation budget.
This should not become another permanent “best quality” switch.
Artificial Analysis measured 87.2 output tokens per second on the Thinking Machines API, but 24.81 seconds to the first answer token, with roughly 22.93 seconds attributed to reasoning. Fast decoding did not mean an immediate answer.
That is the third non-obvious lesson: the effort control belongs beside queue priority, latency budget, and escalation policy. If every request goes to xhigh, the router can erase much of the practical advantage of a 12B-active model.
Classification, extraction, formatting, routing, and short edits. Start at none or low effort, then retry only when validation fails.
Coding, terminal use, tool recovery, and repository changes. Raise effort while keeping tests, permissions, and token ceilings explicit.
Retrieval-backed synthesis over internal material. Require citations and evidence coverage; effort cannot recover a missing source.
Unresolved factual questions, consequential decisions, and long-horizon failures. Change the model or add a human, not merely more thinking tokens.
The production metric should be accepted tasks per dollar and per minute. Track first-answer latency, reasoning tokens, retries, tool failures, human review, and completion quality. Token price alone misses the expensive part of agent behavior.
The weights are labeled Apache 2.0 and published in BF16 and NVFP4. That gives teams real options to inspect, host, fine-tune, and move the artifact. It also reduces the procurement anxiety attached to an API-only model.
But this is better described as managed open than frictionless open.
Thinking Machines attaches a separate Model Acceptable Use Policy to the model materials and modified versions. The policy adds use restrictions, downstream responsibilities, disclosure obligations, and an update mechanism. Teams distributing derivatives or exposing the model to end users should review both documents instead of treating the repository badge as the whole legal contract.
Then there is Tinker. First-party serverless inference costs $0.30 per million uncached input tokens, $0.06 cached, and $1.20 per million output tokens at 256K context. Managed training has different prefill, sampling, and training rates, with a limited-time 50% discount at launch.
The strategic pattern is clear:
open checkpoint lowers adoption risk
managed fine-tuning shortens the learning loop
hosted sampling monetizes iteration
runtime support keeps export credible
Thinking Machines does not need every customer to stay on its inference endpoint. It needs Tinker to become the convenient place where teams teach Inkling-Small their domain. The durable product may be the learning loop—data, renderers, sampling, training, checkpoints, and evaluation—not the raw checkpoint.
That is why this release is more strategically interesting than a cheap API launch. If companies put Inkling-Small into the routine lanes of their agent routers, Thinking Machines can own the adaptation workflow while customers retain a credible exit path.
Inkling-Small accepts text, images, and 16kHz WAV audio and returns text. Thinking Machines recommends image dimensions between 40 and 4096 pixels and audio clips under two minutes. The architecture processes the modalities jointly rather than bolting separate encoders onto a text model.
Provider-reported results include 74.0% on MMMU Pro, 54.9% on Audio MC, 77.0% on MMAU, and 90.1% on VoiceBench. Those numbers make image and audio routing worth testing. They do not establish production parity across modalities.
The model card mentions video in the training-data types, but video is not a documented inference input. Independent testing reviewed for this article does not reproduce the full multimodal suite, long-context behavior, safety behavior, or NVFP4 quality.
Treat each modality as a separate launch:
The right pilot is not a leaderboard replay. It is a small production simulation with enough observability to catch the model's specific failure shape.
Three red flags should stop an early rollout.
First, if a task depends on facts that are not present in the prompt, retrieval corpus, or tools, do not assume extra effort will bridge the knowledge gap. Second, if the business case depends on “12B” meaning cheap workstation hosting, redo the infrastructure model from 276B total and the documented memory floor. Third, if a demo succeeds only at xhigh with a 25-second wait before the answer begins, price and design the user experience around that reality.
Inkling-Small will probably get more production trials than the 975B-total Inkling checkpoint. It keeps almost the same independent composite score while cutting active compute, memory requirements, and first-party output price sharply. Coding, grading, synthetic-data generation, and retrieval-grounded internal assistants are the obvious early targets.
Inference providers will add it quickly because the weights and NVFP4 artifact are public and the vLLM integration is already serious. Their results will diverge. Quantization, context tier, batching, parser behavior, and effort defaults will produce different latency and quality profiles under the same model name.
Agent platforms will begin routing effort and grounding state alongside model identity. “Use Small” will be insufficient. The route will specify low-effort Small, tool-using Small, retrieved Small, or a frontier escalation. This is a cleaner control surface than a ladder built only from model sizes.
Finally, evaluations will have to separate procedural skill from factual coverage. Inkling-Small is a strong counterexample to one-number rankings: near parity on a composite index, better on several reasoning and coding measures, and materially worse on knowledge. Buyers will learn more from a capability profile than a leaderboard position.
The weights are public and labeled Apache 2.0, so “open-weight” is the precise description used here. Thinking Machines also applies a separate Model AUP to the model materials and derivatives. Review both before redistribution or end-user deployment.
Not under the official launch configurations. The provider lists at least 180GB aggregate VRAM for NVFP4 or 600GB for BF16. Community quantizations may explore smaller setups, but they are not the documented deployment floor.
The checkpoint supports up to 1M tokens. Thinking Machines' launch-day managed fine-tuning and serverless inference products expose up to 256K. One-million-token operation is therefore a self-hosted capability that needs separate validation.
It can replace many routine or grounded calls if it passes your repository and tool evals. Keep an escalation path for unresolved factual issues, long-horizon failures, high-stakes actions, and tasks where the cleanup cost dominates token savings.
Reasoning effort. Sweep several effort levels on identical tasks and measure accepted completion, latency, reasoning tokens, retries, and reviewer time. The highest setting is not automatically the best production setting.
Inkling-Small does not collapse the open-versus-closed model debate. It gives builders a better place to draw the boundary.
Closed frontier models can remain on the escalation path for broad knowledge, hard judgment, and failures that deserve expensive recovery. Inkling-Small can take the repeatable, observable, high-volume work where open weights, tunability, private deployment, and cost control matter more.
That split will not happen automatically. The model needs retrieval for knowledge work, bounded tools for agent work, modality-specific evaluation, and a deployment plan based on 276B total parameters rather than the friendlier 12B-active headline.
But if those pieces are in place, the release changes the economics of the agent router. Open weights no longer have to sit outside the serious agent stack as a privacy-only fallback. They can become its working layer.
That is a much bigger role than “small model.”