OpenAI’s Codex Security CLI Makes Missing Evidence Fail the Build
OpenAI’s Codex Security CLI Makes Missing Evidence Fail the Build
OpenAI has open-sourced the Codex Security CLI and TypeScript SDK. The obvious reading is that OpenAI has released another AI vulnerability scanner. That misses the useful part.
The important release is a security-review state machine: it records what the agent inspected, what it deferred, which findings survived validation, whether an old finding genuinely disappeared, and what CI should do when evidence is incomplete. Teams can inspect and extend that workflow. The reasoning model remains hosted, access-controlled, and tied to OpenAI credentials.
OpenAI is putting the orchestration layer in public while keeping the capability layer behind an authorization gate. Builders get typed findings, local history, SARIF export, patch workflows, cost callbacks, and explicit CI semantics. They do not get an offline model or proof that this replaces CodeQL, SCA, secret scanning, or human AppSec review.
The sharpest product decision is buried in the exit codes. A scan with partial or unknown coverage returns 2, even when it found no vulnerability above the configured threshold. In other words, “nothing found” is not allowed to masquerade as “nothing exists.” For nondeterministic security agents, that may matter more than one more benchmark claim.
What OpenAI actually put in public
The public repository contains @openai/codex-security, a CLI and server-side TypeScript SDK released under Apache 2.0. Version 0.1.0 reached npm on July 28, 2026; 0.1.1 followed later that day. OpenAI formally announced the release on July 29 and called it an early, access-gated beta.
The package can review a full repository, selected paths, a committed diff, or staged and unstaged changes. It supports standard and deep review, validation, patch proposals, scan history, bulk campaigns, and JSON, CSV, or SARIF exports. Teams can also supply architecture documents, threat models, and security policies.
That is not a thin npx wrapper around a prompt. The repository bundles schemas, workbench scripts, integration code, and thirteen named security skills covering the path from threat modeling and discovery through validation, triage, patching, reporting, and tracking. This connects directly to the broader move I covered in OpenAI Skills Turns Agent Workflows Into a Cross-Product Standard: the durable artifact is increasingly the workflow contract around the model.
But “open source” needs a precise label here.
| Layer | What is public | What remains controlled | Builder consequence |
|---|---|---|---|
| Workflow | CLI, SDK, skills, schemas, artifact handling, CI examples | Hosted product services and some validation infrastructure | You can audit and embed the control flow |
| Reasoning | Model configuration is visible in source | Model weights and inference remain hosted | This is not an offline or provider-independent scanner |
| Access | Package and source are publicly readable | Scans require beta entitlement; some work may require Trusted Access for Cyber | A successful install does not prove authorization |
| Evidence | Findings, coverage, manifests, reports, exports, local history | Detection quality still depends on model behavior and product policy | Auditability improves; determinism does not magically appear |
The new artifact is a loop: target and context become findings plus coverage, which become validation, comparison, remediation, and the next scan’s evidence.
Coverage is the product’s most important output
Traditional CI encourages a dangerous simplification: green means good, red means bad. An agentic scanner needs a third state—the analysis was not complete enough to support either conclusion.
Codex Security makes that state explicit. Its coverage document records reviewed surfaces, exclusions, deferred work, open questions, and a completeness value of complete, partial, or unknown. The official exit-code contract then separates outcomes:
This is OpenAI’s answer to a reliability problem.
An LLM-driven review can vary between runs. It can run out of budget, defer a path, lose an environment dependency, or fail during validation. If CI collapses those cases into “no high-severity finding,” teams turn missing evidence into a clean bill of health. Codex Security refuses that shortcut.
The same rule carries into history. According to the CLI FAQ, comparison can label a finding new, persisting, reopened, resolved, or unknown. A finding is resolved only when the later scan covers the original target and affected path. Failure to rediscover an issue is insufficient.
My first RohitAI read is that coverage-aware failure will become standard across agentic security tools. Scanner vendors can argue about models and benchmarks. Buyers will eventually demand a machine-readable answer to a simpler question: what did you actually inspect before you declared this build safe?
The CLI is a findings workbench disguised as a scanner
A stateless scanner takes code in and emits alerts. Codex Security keeps a ledger.
Completed scans produce scan-manifest.json, findings.json, coverage.json, report.md, supporting artifacts, and optional exports. The workbench stores scan history and supports rerun, match, and compare operations. False-positive feedback can inform future scans, but OpenAI says it does not permanently suppress a rule, path, or vulnerability class; the current source is checked again.
This changes the unit of value. It is no longer only “did the model spot a bug?” It is:
- Can the team establish that the same finding persists across revisions?
- Can reviewers see why a result was validated or dismissed?
- Can a later scan distinguish “fixed” from “not re-examined”?
- Can the organization replay what model, policy, and repository state produced the result?
That evidence graph may be more defensible than raw detection performance. Models will change. A durable record of scope, identity, reviewer judgment, remediation, and revalidation is what turns a demo into an operational system.
OpenAI’s own product story has shifted in this direction. Its March research-preview announcement emphasized more than 1.2 million commits scanned in 30 days, 792 critical findings, 10,561 high-severity findings, and claimed reductions in noise and false positives. Those are OpenAI-reported figures from an earlier cloud cohort, not an independent benchmark of CLI 0.1.1. By June, OpenAI was emphasizing findings marked fixed across tens of thousands of codebases. The July package makes that remediation ledger portable.
This is closely related to the argument in OpenAI’s Long-Horizon Agent Failures Make the Session the Security Boundary: once an agent performs extended work, its state, evidence, tools, and interruptions become part of the security boundary. Codex Security applies that principle to AppSec itself.
The scanner has its own threat model
OpenAI’s reference GitHub Actions workflow contains a quiet warning: the thing inspecting hostile code is privileged infrastructure.
The guide installs an approved Codex Security package under the runner’s temporary directory before checking out pull-request code. It runs the scanner by absolute path, disables persisted checkout credentials, scopes the API key to the scan step, writes output outside the worktree, and pins Actions by commit. The example also skips forked pull requests and Dependabot because normal secrets are unavailable or restricted there.
That ordering is not ceremony. A branch under review must not be able to replace the executable, dependency source, configuration, or script that receives the scanning credential. The scanner sees proprietary code, threat models, vulnerability evidence, and an API key. Validation may execute tests or proof-of-concept behavior. A malicious repository can attack the reviewer.
There is a second governance issue hiding in the knowledge base. Architecture files, SECURITY.md, threat models, and policy documents influence what the agent treats as trusted, exposed, severe, or out of scope. That makes them executable security inputs. Put them behind CODEOWNERS or an equivalent approval rule. Otherwise, a code change can quietly rewrite the assumptions of the agent assigned to review it.
The skipped-fork path also needs an honest label. The reference workflow avoids exposing secrets to untrusted contributors, which is correct. It also means external and automated dependency changes do not receive this exact scan. Teams need a separate trust-safe route—a sandboxed merge queue, an internal mirror, or a credential broker—not a dashboard that silently implies every pull request was covered.
Reproducibility requires more than pinning one npm version
At publication time, npm’s latest tag is 0.1.1. The package pins @openai/codex and @openai/codex-sdk to 0.144.6, while the bundled Codex Security plugin reports version 0.1.14. The hosted catalog and public plugin marketplace have also carried different plugin versions, and OpenAI’s plugin changelog warns that rerunning a saved scan does not pin the installed plugin.
The public SDK configuration currently defaults to gpt-5.6-sol at xhigh reasoning with native multi-agent v2 and as many as nine concurrent threads per session.
That creates a provenance problem. “Scanned with Codex Security 0.1.1” is not enough to reproduce a result. The model, reasoning effort, plugin, Codex runtime, target revision, knowledge base, policy, credentials, environment, and coverage can all affect the answer.
This is my second RohitAI read: AI scan provenance will become a procurement and compliance requirement. Security teams already ask which ruleset and database version produced a deterministic result. An agentic result has more moving parts, so its manifest needs to be richer, not looser.
Cost turns review depth into a scheduling decision
The default configuration is not a lightweight linter. GPT-5.6 Sol at xhigh effort with up to nine concurrent threads is a multi-agent workload. The SDK exposes maxCostUsd and an onCost callback, but OpenAI’s documentation says the cost limit is estimated rather than hard; requests already in flight can finish above it.
OpenAI has not published a Codex Security-specific per-scan price, quota, latency target, SLA, or general-availability date. The default model’s list price is therefore context, not a reliable scan quote. As I argued in OpenAI’s GPT-5.6 Codex Rate Card Makes Agent Budgets a Routing Problem, agent cost is shaped by orchestration, retries, context, caching, and completion—not one token number.
Most teams will end up with scan tiers:
Scan the exact committed change, retain coverage and findings, measure latency, and avoid blocking until the team understands signal quality.
Revisit the full repository or high-risk paths on a cadence, compare against prior state, and feed accepted context back into triage.
Reserve broader reasoning and validation for sensitive releases, authentication changes, exposed services, or periodic security campaigns.
This tiering gives each result the right expectation. A PR diff scan answers a change-level question. A repository scan searches for broader attack paths. A deep review trades more time and spend for wider reasoning. Mixing those outputs into one “secure” badge would erase the scope evidence that makes the tool useful.
A rollout that can survive contact with a real repository
The sensible starting point is not “replace the scanners.” It is “measure a new contextual layer beside them.”
GitHub already applies CodeQL, dependency checks, and secret scanning to code produced by third-party coding agents, including Codex. Codex Security is better positioned as a complementary research, validation, and remediation layer: it can reason from architecture and threat models, trace multi-file behavior, pressure-test candidates, and propose system-aware patches. Deterministic tools still provide repeatable rules, dependency intelligence, secrets coverage, and mature policy controls.
Start with representative repositories, not a polished demo. Include an old monolith, a modern service, a monorepo, uncommon frameworks, and a project with painful setup. OpenAI has not published a current language, framework, build-system, monorepo-size, or operating-system support matrix for the July CLI.
What the release does not prove
The repository improves auditability, not evaluation.
There is no independent head-to-head benchmark of CLI 0.1.1 against CodeQL, Semgrep, Snyk, GitHub Code Quality, Claude Code Security, or specialist agentic security products. OpenAI’s earlier claims—such as noise falling 84% on one repository and false-positive rates falling by more than 50% across its beta repositories—lack enough public methodology to predict results on a buyer’s code.
The March and June commit and remediation figures describe earlier cloud and plugin deployments. They should not be cited as measured throughput, recall, or cost for the July open-source package.
Access is also fragmented by surface. The cloud research preview has documented ChatGPT plan availability, while the CLI and SDK are for approved beta customers and partners. The SDK guide adds that some full-repository scans may require Trusted Access for Cyber, which an API key alone does not grant. Public source does not mean public capability.
Finally, repeat scans can differ. Finding matching and coverage rules make that variation governable; they do not make the model deterministic. The right buyer question is not “does it always produce the same alert list?” It is “does the system preserve enough evidence to explain, review, and safely act on the differences?”
RohitAI’s read: the evidence layer is the long game
Three things are likely to follow.
First, coverage-aware CI will spread. Agentic review cannot credibly use binary pass/fail semantics when an interrupted or budget-limited run is possible. Vendors will expose separate states for policy failure, execution failure, and insufficient evidence.
Second, shared findings memory will become a product battleground. Codex Security now has local history, stable identities, false-positive context, cloud threat models, and multiple surfaces. The obvious missing layer is governed synchronization across CLI, CI, plugin, and cloud: ownership, retention, conflict resolution, portable decisions, and team-level audit.
Third, the SDK may matter more commercially than the CLI. Typed findings, lifecycle callbacks, cost events, coverage, and artifact paths make Codex Security embeddable inside developer platforms, AppSec products, and internal portals. OpenAI can supply the evidence engine while partners own the workflow.
The competitive moat will not be “our model found a vulnerability once.” That capability will diffuse. The moat will be a trustworthy loop:
scope -> inspect -> explain coverage -> validate -> assign
-> patch -> recheck -> preserve evidence -> learn
That is why this release matters. OpenAI is moving Codex from producing code into governing the evidence around code. The company has not open-sourced the intelligence. It has open-sourced a serious attempt at making that intelligence operational.
FAQ
Is the Codex Security CLI available to everyone?
No. The source repository and npm package are public, but OpenAI says the CLI and SDK are in beta and require access. Some full-repository scans may additionally require Trusted Access for Cyber.
Does open source mean scans run locally without OpenAI?
No. The workflow code is open under Apache 2.0, but scanning still uses Codex, OpenAI authentication, and hosted inference.
Can Codex Security replace CodeQL, SCA, or secret scanning?
There is no evidence for that claim. Use it as a contextual research, validation, and remediation layer beside deterministic scanners and human review until representative testing proves where it adds value and where it misses.
What does exit code 2 mean?
It covers input, runtime, export, or bulk errors and scans with partial or unknown coverage. Treat it as “no reliable security conclusion,” not as a clean result.
Does --max-cost guarantee a hard spending cap?
No. OpenAI describes the limit as an estimate. In-flight requests may finish above it, though available partial results are preserved when the limit stops a scan.
What should a team save after each scan?
Keep the manifest, findings, coverage, report, supporting artifacts, relevant exports, exact target revision, configuration and plugin provenance, reviewer decisions, and remediation status. SARIF alone is useful for triage but does not preserve the full evidence contract.
Final take
OpenAI’s Codex Security release deserves attention because it refuses two convenient fictions.
The first is that public orchestration code equals an open security model. It does not. This is an auditable, extensible control plane around a hosted and access-controlled capability.
The second is that an empty alert list equals safety. It does not. A useful agentic security system must say what it covered, what it deferred, and whether a later run truly revisited the evidence behind an old finding.
Add Codex Security as a measured layer. Protect the scanner from the code it inspects. Version the context that shapes its judgment. Preserve the complete evidence bundle. Keep deterministic tools. Let advisory data earn the right to block a merge.
If OpenAI gets the finding lifecycle right, the long-term product will not be another scanner. It will be the system that remembers why a codebase was considered risky, what changed, and whether the evidence is finally strong enough to say the risk is gone.