IBM’s October 27, 2026 ScarfBench report gives businesses commissioning AI-assisted Java migrations a reason to look beyond a successful build. Across 204 migration tasks and ten model-and-agent configurations, the highest reported rate of passing every test was 72.1%.
For the person signing off the work, the useful question is whether the business functions still work after the software foundation changes. A migration acceptance plan should make that question explicit before anyone starts changing code.
ScarfBench was introduced in March. IBM now says Kaggle reproduced the evaluation and will maintain the leaderboard; public model submissions are unavailable at launch.
What a successful build leaves unanswered
A framework supplies the software foundation and conventions an application relies on. ScarfBench covers moves between Jakarta EE, Quarkus and Spring, with separate checks for building, starting and preserving behavior. Compilation establishes that the code can be built. Startup establishes that it can launch in its target environment. Neither alone establishes that a user can complete the intended task.
IBM reports an address-book migration in which pages for listing and creating contacts returned HTTP 404—page-not-found errors. It passed two of ten checks. This was a benchmark example, not a customer incident.
The address-book example includes creating, editing, listing and viewing contacts. For an owner commissioning a similar migration, those operations provide a concrete starting point for acceptance: can someone add a contact, find it again, change a detail and retrieve the changed record? A page that opens is not enough if its saved information is wrong.
Two scores answer different questions
On the full 204-task set, Gemini 4 Argon with Antigravity scored 75.2% on the weighted behavioral-test measure, called the smoke score, versus 70.5% for Claude Opus 5.5 with Claude Code. Kaggle’s public results export confirms those two scores. Separately, IBM reports that Claude Opus 5.5/Claude Code led the all-tests-pass measure at 72.1%, against 71.1% for Gemini/Antigravity.
The weighted score awards partial credit and gives larger test suites more weight. All-tests-pass counts tasks with every check passing. These are 204-task results, not the report’s separate 200-task sensitivity analysis, which excludes four tasks. Kaggle’s export does not include the all-tests-pass measure.
The practical distinction is between ranking a tool and accepting a deliverable. Partial credit helps compare how much of a benchmark each configuration completed. It cannot decide whether a missing function is tolerable in your application. A contact system that cannot create contacts should not be accepted simply because other checks lifted its average.
Agree the evidence before approving the migration
Start with the business operations that must survive, not a target percentage copied from a leaderboard. The ScarfBench paper defines scenarios through an initial state, an operation and an expected result at the application’s external interface. That suggests an acceptance specification written around what the software must do, rather than how its old internal classes were arranged.
For a migration you commission, request three separate results:
Build: A repeatable build of the exact code revision being reviewed, with the target dependencies identified. A successful build of an earlier revision is not evidence for the final changes.
Run: Evidence that the application starts in the intended target environment with the databases or other services it needs. Keep startup failures distinct from failures in the user’s workflow.
Preserve the task: Results for each agreed essential operation, showing the starting data, expected response or stored outcome, and actual outcome. Include failed checks, not just a summary percentage.
Treat these as proposed acceptance requirements, not a claim that ScarfBench certifies your application. Agree any intentionally changed behavior explicitly; otherwise a missing feature can be mistaken for part of the redesign.
Keep the model and its surrounding coding tools identifiable too. Kaggle’s entries name agent versions and reasoning settings, while the ScarfBench quickstart documents saved inputs, outputs, logs and run metadata. Request the model, agent version, relevant settings, source and target framework versions, generated changes and validation logs. Buying access to the same model does not recreate an entire evaluated setup.
For a related but different upgrade problem, RohitAI’s Claude API compatibility guide explains checks across requests, connections and business results.
Passing the tests is still a bounded result
The paper’s limitations matter at sign-off. Its checks cover observable behavior, not every internal defect: race conditions, resource leaks and performance regressions can escape them. The authors provide no formal coverage guarantee and acknowledge that public benchmark applications may overlap with model training data.
For your application, add separate evidence for security, performance, concurrent use and operational requirements where they matter. The 204 tasks come from 34 application families across three frameworks, according to the paper; they are not 204 independently sampled customer projects. A benchmark percentage is therefore not a forecast of your own migration’s chance of success.
Use the results to shortlist a setup for a representative pilot. Make the release decision against the functions your workers or customers need, with unresolved failures visible to the person accepting the work.