Benchmarks are a real signal of capability - and easy to misuse. This page explains how we measure autonomous coding agents, what the standard benchmarks do and do not capture, and how CodeCourier reports results: only numbers we can reproduce and stand behind.
We evaluate against two public, respected benchmarks. Each tests something real about an agent's ability to do engineering work, and each has limits worth understanding before reading any score.
A human-validated set of real GitHub issues from open-source projects. An agent is given the issue and the repository and must produce a patch; the task counts as solved only if the change makes the project's own tests pass. It measures whether an agent can fix real bugs in real code, not whether it can ace a quiz - the closest thing the field has to a meaningful exam for an AI software engineer.
A benchmark for agents that work through a terminal - running commands, inspecting output, and iterating toward a goal in a real shell environment. It complements SWE-bench by testing operational, tool-using competence rather than only patch generation, which matters for an agent that has to set up, build, and verify its own work.
Any figure we publish comes with the exact methodology, the date, and the scaffold used to produce it. If a result is not reproducible, it is not on this page.
A score reflects a whole system - how the agent gathers context, plans, and iterates - not just the underlying model. Two products on the same model can score very differently, so a number describes a system, not a model.
Pass@1 versus multiple attempts, which subset, which date, what tooling - all change the number. We compare like with like and label the conditions, or we do not compare.
We are deliberate about benchmarks. Rather than lead with a headline percentage, we publish verified, reproducible results here as our runs complete - each with its full methodology, date, and scaffold so anyone can check it. This section lists the categories we report against; the numbers land here as they are measured and independently verifiable.
We have chosen not to print a benchmark score we cannot yet reproduce and independently verify. Publishing an unverified number would be marketing, not measurement. The table below shows the categories we will report - every status reads pending verification, and nothing here is a measured result. When a figure is reproducible, it appears here with its full methodology.
This table is methodology, not measured results. Rows are the categories we report against; no row contains a score. Verified figures are added here as runs complete and can be independently reproduced.
Capabilities move weekly and scores go stale fast, so we do not freeze competitor percentages into this page. For current standings, go to the source: the official leaderboard and each vendor's own dated, published figures.
The canonical public ranking of agents on SWE-bench and SWE-bench Verified, maintained by the benchmark's authors. It is the right place to see live, current standings rather than a number copied into an article that will be wrong next month.
Open the SWE-bench leaderboardFor any specific agent - Devin, Claude Code, OpenAI Codex, Cursor, GitHub Copilot, OpenHands and others - check that vendor's own site for their latest, dated figures and methodology. We name competitors only to compare fairly, and we do not quote percentages we cannot verify.
As of June 2026. Treat any benchmark figure - here or anywhere - as a dated snapshot, and check the live leaderboard and each vendor's site for current numbers.
5 minutes to onboard. First PR within an hour. Cancel anytime.