A coding agents benchmark run by Composio across 30 real-world scenarios puts Claude Code first on execution speed, at 122 seconds per task. Each successful task costs $0.195 there, close to three times the price of OpenCode.
Key Takeaways
- Four agents compared on the same 30 Gmail, GitHub, Slack and Notion tasks, all running on DeepSeek V4 Flash.
- Claude Code finishes a task in 122 seconds against 272 for Oh My Pi, a 2.2x spread.
- Seven tasks passed or failed purely on which framework ran them, with overall success rates staying comparable.
Have an AI Sum Up This Article
ChatGPTFour agents, a single model underneath
The protocol Composio picked to compare these coding agents has one clear merit: it removes the model variable. All four agents compared, Claude Code, Codex, OpenCode and Oh My Pi, ran on DeepSeek V4 Flash.
That choice isolates what the test is actually measuring. Model reasoning quality is not on trial here, the way each framework splits a task, calls its tools and manages its context is.
Most published agent comparisons skip that step and swap models between contenders, which makes the results almost unreadable. Holding the model constant is what turns a marketing chart into something an engineering team can act on.
The question matters more because access to these tools is far from guaranteed inside a company. Several large groups have restricted specific agents internally, as when Alibaba blocked Claude Code for its own employees. Knowing what one framework genuinely adds over another turns into a concrete procurement call.
The 30 scenarios run on tools everyone touches daily: Gmail, GitHub, Slack and Notion. No academic puzzle, no synthetic benchmark, just chains of the read a thread, open an issue, post a message, update a page variety.
Thirty scenarios is a modest sample, and worth keeping in mind before treating any of these numbers as settled. It is enough to expose a pattern, not enough to certify one framework over another for a given organisation.
That terrain choice counts. These tasks go through third-party APIs, with authentication, pagination and inconsistent response formats along the way. Orchestration quality shows up there far more clearly than on an isolated coding exercise.
Success rates stay comparable across agents, which makes the rest of the numbers readable. Oh My Pi clears 17 tasks out of 30, OpenCode manages 14, and the spread between the extremes is not wide enough to separate frameworks on reliability alone.
The most uncomfortable finding sits elsewhere. Seven tasks flipped from success to failure, or the other way round, purely on which framework carried them. Same model, same instruction, opposite outcome.
Seven out of thirty is close to a quarter of the set. At that rate, a headline success percentage stops being a reliable summary of anything, because the number would move on a different task list drawn from the same tools.
122 seconds against 272, for $0.195 against $0.073
On speed, the gap is clean. Claude Code runs a task in 122 seconds, Oh My Pi needs 272, a 2.2 ratio between the fastest and the slowest in the panel.
On cost, the ranking flips. A successful task comes in at $0.195 on Claude Code and $0.073 on OpenCode, a factor close to three between the most and the least expensive of the four.
The reason is not token waste. Claude Code makes the fewest tool calls in the panel and generates the fewest output tokens. It does less, faster, and still costs more.
That apparent contradiction points at billing structure rather than technical frugality. A framework that is sparing with calls can stay expensive if the unit price of what it consumes runs higher, and that is precisely the trade-off this benchmark documents.
None of this is new for teams already shipping. We ran a file on it when coding agent invoices started seriously worrying engineering budgets, and these measurements finally attach comparable numbers to that worry.
The amounts also need scaling to something realistic. Twelve cents of difference per task stays trivial on a single developer workstation. Across a continuous integration pipeline firing several thousand runs a month, the same gap turns into several hundred dollars.
One caveat deserves stating. These figures hold for one model, on one date, across one specific task set. A pricing change on the vendor side is enough to reshuffle the whole ranking without a single line of framework code moving.
More articles on Horizon
- Coordinated Breaches Force OpenAI to Slow Research
- Anthropic AI Chip Plans Get Their Own Team
- DeepSeek Pricing Is About to Jump Sharply
Which trade-off suits which workload
For a team, the question becomes easy to phrase: what costs more, two minutes of waiting or twelve cents on the invoice? The answer depends entirely on where the agent runs.
In interactive use, a developer sitting there waiting for the agent before resuming work pays back the price gap many times over. Two extra minutes per task, multiplied across dozens of tasks a day, costs far more in human time.
Waiting also has a hidden cost nobody puts on an invoice. Past a certain delay the developer switches to something else, and the context reload when the result finally lands eats more minutes than the wait itself.
For background processing, the trade-off reverses completely. An overnight task queue has no reason to pay three times more for a result delivered earlier in a night nobody is watching.
Nothing forces a single choice either. A team can route interactive tasks to the fastest framework and leave deferred processing on the cheapest, provided it accepts the maintenance load of two separate integrations.
The volatility measured on seven tasks changes the nature of the decision though. A framework that makes a task pass or fail independently of the model forces you to test on your own corpus before choosing, because a panel average says nothing about your particular case.
That volatility argues for running two frameworks side by side through the qualification phase. Paying twice on a few dozen representative tasks is negligible next to the cost of industrialising the wrong choice for a year.
Anthropic kept widening its agent’s scope through the summer, notably when Claude Code gained a built-in browser to act directly on the web. Every added capability shifts the comparison and dates the previous benchmark.
For everyone building coding agents the message reads clearly. The differentiator no longer sits in raw ability to complete the task, since the model is interchangeable, it sits in orchestration efficiency. That is ground an open project can make up in a matter of months.
Follow the story on Horizon.
