Across six runs of Andon Labs’ Vending-Bench 2, the Astra agent closed with an average of 15,515 dollars in the bank against 5,422 for Claude Fable 5.1, and its worst run still beat its rival’s best. The same model became the first to clear the human baseline on all five Drone-Bench subtasks, at least once on each.
Key Takeaways
- None of the six vending runs from OpenAI’s model dropped below Claude Fable 5.1’s best result.
- Fable 5.1 lost 14,331 dollars in prepayments to suppliers that never delivered, against zero identified losses on the other side.
- On the drone, a full mission still lands only 2.8 percent of the time on an average run.
Have an AI Sum Up This Article
ChatGPTSix Vending Runs the Astra Agent Never Lost
Andon Labs has spent a year running an evaluation that hands a model the full operation of a vending machine. The model orders stock, negotiates with simulated suppliers, sets prices, handles refund requests and lives off its own cash position. The latest campaign, published on September 7, gives an average closing balance of 15,515 dollars to the Astra agent against 5,422 for Claude Fable 5.1.
The spread matters more than the average. Over six games each, the worst run from the Astra agent still lands at 13,272 dollars, while Fable 5.1 tops out at 9,874. The two series stay apart the whole way, Anthropic’s model holds second place in every game, and this is the first time an OpenAI model has topped this board.
The gap traces back to one mistake OpenAI’s model never makes. Fable 5.1 wired 14,331 dollars of advances to suppliers that never shipped, which works out to 2,389 dollars burned per run. The Astra agent hit 64 supplier closures across its own games with no loss of that kind on record.
Purchase discipline tells the same story. The average price Fable 5.1 paid for a 12-ounce can drifts from 1.17 to 2.21 dollars as orders pile up, while the Astra agent holds between 1.05 and 1.15. Andon Labs logged one negotiation where the model held its ground at 108 dollars against an opening quote of 226.32.
Customer handling splits the two far less cleanly. The Astra agent paid out 230 refund requests out of 240, Fable 5.1 paid 224 out of 237. That difference sits just above one point, nowhere near the cash gap.
What the protocol rewards is worth stating plainly, because it is not intelligence in the abstract. A run is scored on the cash left at the end, after every ordering decision, every price change and every refund call has compounded. Six games per model give enough repetitions to separate a lucky streak from a habit, and the habit is what the balances expose.
Andon Labs also ran three head-to-head games with Astra, Fable 5.1 and GLM-5.3 trading on the same market. OpenAI’s model won all three, averaging 12,363 dollars ahead of GLM-5.3 at 7,841 and Fable 5.1 at 5,719. During those games the Astra agent turned down the price-fixing offers GLM-5.3 put on the table, and Fable 5.1 took them.
The result extends a tight run of announcements. OpenAI claimed the AGI era had started when the model shipped in early September, a claim that called for hard numbers from someone else. Here is one set, produced by an outside lab on a public protocol.
The Drone That Follows a Person Around an Office
Andon Labs’ second evaluation leaves simulation behind. Drone-Bench hands a model the controls of a real aircraft inside real offices, across five subtasks that run from spotting a person to rebuilding a room in three dimensions.
The Astra agent is the first whose best attempts clear the baseline set by a human-assisted team, and it does so on all five subtasks. Andon Labs published a clip where the model is told to find a specific person and follow them, then navigates the lab’s own offices on its own.
Success rates pull the demo back to earth. Person detection lands on 4 runs out of 10, three-dimensional reconstruction on 1 out of 10. The model gets there once and then fails to repeat it on command. Raw capability was settled elsewhere over the summer, when OpenAI’s model cracked ten open math problems.
The baseline being cleared here deserves a second look too. It was not set by a person flying manually but by a human team working with AI assistance, which makes it a moving target rather than a fixed ceiling. Beating it once on each subtask is a first, and it is still a long way from beating it on demand.
Chaining all five steps in order on a single attempt stays the exception, at 2.8 percent of average runs. A capability that fires three times out of a hundred is worth something very different from a capability that is simply available, and that reliability is what still separates the bench from the field.
Andon Labs puts a date on the rest. Frontier models could solve every task in a single attempt by the first quarter of 2027, the lab projects. That estimate is the lab’s own and rests on no vendor roadmap. Artificial Analysis had already rebuilt its ranking in early September to absorb that kind of gap between claim and measurement.
OpenAI’s internal trajectory makes the projection less abstract than it sounds. The productivity gain the company measured on its own teams had already pulled its schedule forward by six months, which mechanically shortens the gap between two generations.
More articles on Horizon
- OpenAI Asks Congress If Slowing AI Down Is Legal
- Claude Max Buyers Sue Anthropic Over Usage Limits
- DeepSeek V4.1-Flash Cuts Agent Memory Eightfold
What the Jump Forces on Deployers and Rival Labs
For a product team weighing whether to hand an agent a real budget, the numbers move the question. Capability is no longer the blocker, variance is. An agent that prints money half the time and torches the float the other half is unusable, whatever its average score says.
Prepayment is the most transferable test in the whole protocol. An advance sent to a supplier that goes dark shows up in any automated purchasing flow, and that is exactly the line where the two models diverge by 14,331 dollars.
Turning down price fixing speaks to another function entirely. A compliance team looks first at what the Astra agent, or any model in its seat, agrees to when a competitor offers an arrangement, and Andon Labs produces that answer under controlled conditions.
On the competitive side, Anthropic loses on this bench the argument that has framed its messaging for two years. The model that wins is also the one that refuses the cartel, so performance and alignment no longer pull apart.
GLM-5.3 slots between the Astra agent and its Anthropic rival, and that detail matters to buyers. A Chinese model that doubles Fable 5.1 on a third-party protocol makes the pricing case harder to defend internally, especially for repetitive agent workloads.
One number is missing to close the analysis. Andon Labs publishes balances and keeps the compute each game burned to itself, while the ratio between the two decides whether an agent actually pays for itself. Our own test priced what the generational jump costs in daily use, and that line weighs as much as raw performance.
Buyers running pilots right now get a cheap lesson out of all this. Watching an agent handle a supplier that disappears, or a rival that offers to rig prices, says more about production readiness than any leaderboard position, and both scenarios can be staged internally in an afternoon.
The next signal to watch is Anthropic’s answer on the same protocol. A patched Fable 5.1 that stopped losing advances would claw back most of the gap, and the company has the detailed logs it needs to try.
Follow the story on Horizon.
