On October 2, 2026, ServiceNow CoreAI described AutoSynthData, a pipeline for turning an enterprise agent’s weaknesses into training data. AutoSynthData generates new tasks, checks their solutions and verifiers, then uses accepted samples for post-training. For teams adapting models to company workflows, the key requirement is an environment where success can actually be checked.
This is a method to adapt, not an installation guide for a released AutoSynthData package. As of October 2, the announcement links the existing EnterpriseOps Gym benchmark and base-model cards, but no generator implementation, synthetic training corpus, fine-tuned checkpoint or price list.
What the reported gains establish
ServiceNow reports supervised fine-tuning (SFT) of Gemma-4-26B-A4B-it in two EnterpriseOps Gym environments: Hybrid, covering multi-domain work, and ITSM, or IT service management. Its published results give the following comparison. These are company-reported measurements, not an independent reproduction.
Environment | Training samples | Mean Pass@1: before → after SFT | Gain, calculated |
|---|---|---|---|
Hybrid | 2,000 | 20.45% → 27.65% | 7.20 percentage points |
ITSM | 1,994 | 18.77% → 27.18% | 8.41 percentage points |
Calculation: Hybrid’s gain is 27.65 − 20.45 = 7.20 percentage points; dividing 7.20 by the 20.45 baseline gives about 35.2% relative improvement. The distinction matters: both trained task-success rates remain below 28%. The results support investigating targeted training, not assuming an agent is ready for unattended production work.
Do not substitute the higher verifier score for whole-task success. ServiceNow separately reports Hybrid verifier success reaching 68.55%. The benchmark’s scoring documentation distinguishes tasks satisfying every condition from the average pass rate of individual conditions.
The announcement does not specify evaluation denominators, uncertainty intervals or whether checkpoint selection used an untouched validation split. It establishes neither production transfer nor a controlled comparison with other ways of generating training data. The reported experiments use SFT; reinforcement learning remains a proposed extension.
A pilot workflow for enterprise teams
The following is proposed engineering guidance derived from the method, not a tested recipe or an official AutoSynthData interface. Start with one workflow whose state, permissions and expected outcomes you can inspect.
1. Make the environment repeatable
Use an isolated simulator or test environment, not live enterprise records. Identify the allowed tools, policy constraints, starting data and outcomes you can verify. A failed run is not evidence of a model weakness if the requested action was impossible or the required tool was unavailable.
The public EnterpriseOps Gym repository illustrates the infrastructure: seeded SQL snapshots, containerized tool servers and checks of final database state. Its benchmark runner does not implement AutoSynthData generation or fine-tuning. For your pilot, pin code, dataset revisions, container digests and model settings so a changed environment does not masquerade as a training gain.
2. Separate diagnosis from final evaluation
Before generating examples, reserve a final test set that will not shape capability selection or checkpoint choice. Use a separate diagnostic set to compare the target model’s failures with a stronger teacher’s successful runs.
ServiceNow says its capability cards describe skills and workflow requirements without passing original evaluation prompts, entities, trajectories or verifier details to the generator. That reduces direct copying; it does not make evaluation independent of curriculum design, because evaluation failures still guide what gets generated. Nor is removing those fields a formal privacy guarantee.
3. Keep a task package, not just a prompt
The source’s task definition combines system constraints, a user request and a verifier. A practical manifest for your own implementation should retain:
The initial-state identity, permitted tools and policy version.
The request, targeted capability and seed-task lineage.
The reference execution, verifier version and validation outcomes.
The generator, teacher and target-model identities and inference settings.
This is a suggested record structure, not a published AutoSynthData schema. Keeping these pieces together lets reviewers distinguish a bad demonstration from an impossible task or an incorrect success check.
4. Test what the verifier accepts and rejects
AutoSynthData’s positive and negative checks execute a reference solution and challenge the verifier with incorrect outcomes. A successful reference shows that one solution works. It does not show that every valid solution will pass, or that every invalid one will fail.
Consider an illustrative, unexecuted ITSM task: reassign one uniquely identified, approved incident to a specified team while leaving protected records unchanged. The proposed checks below turn that request into reviewable acceptance criteria.
Candidate outcome | Expected result | What the check probes |
|---|---|---|
The intended incident is reassigned; protected records are unchanged. | Pass | The reference solution works from the stated starting state. |
A different incident changes; the intended incident stays unchanged. | Fail | The verifier checks identity and the requested field, not merely that an update occurred. |
The intended incident is reassigned, but an unrelated protected record also changes. | Fail | The verifier enforces the no-side-effects constraint. |
A different policy-permitted action order reaches the correct outcome. | Pass | The verifier accepts valid alternatives, not only one reference sequence. |
Reset state between trials. If policy constrains intermediate actions, retain an action trace as well as the final state. A finite collection of wrong-outcome checks tests those cases only; separately review missing constraints and alternative valid paths. This parallels the distinction between passing checks and broader validity in our BootLoops verification guide.
5. Expand only after candidates pass
In its reported difficulty filter, ServiceNow favors tasks solved by the target on at most one of three trials and by the stronger solver on at least two of three. Treat this as a sampling heuristic, not a precise estimate of difficulty.
Bound repair attempts and rerun checks after changes. Review rejected candidates and coverage by workflow family, not just the accepted count. ServiceNow’s expansion method also requires full validation of variants and prevents a variant from seeding another variant. Preserve that lineage if you adopt the pattern.
6. Decide whether training earned its cost
Keep a baseline checkpoint and compare it with the trained candidate under the same tools, environment revision, inference budget and scoring rules. Select checkpoints on validation data; use the reserved test set for the final comparison. Report task success, individual-condition results, policy violations, errors and denominators separately, including regressions on workflows the model already handled.
Budget generation, rejected candidates, solver trials, repairs, review and training. ServiceNow reports about 18 hours to generate Hybrid’s 2,000 samples and 66 hours for ITSM’s 1,994. Those runs used different teachers and pipeline stages; they are not a controlled speed comparison. Hardware, concurrency, monetary costs and training duration are not supplied, so the figures cannot establish cost per usable example.
A sensible first decision is whether you can reset the workflow and reliably distinguish acceptable from unacceptable outcomes. If not, build those checks before scaling generation. If you can, compare a bounded SFT pilot with using a stronger model directly, accounting for both operating cost and the cost of maintaining the training pipeline.
Methodology: This AI-assisted guide draws on ServiceNow’s published announcement and commit-pinned benchmark documentation. Calculations reuse ServiceNow’s reported measurements. No model inference, training, production evaluation or execution of the illustrative checks was performed.