Anthropic published its first detailed measurements of how much of its own research is automated on September 17: in August, Claude led 26% of model R&D tasks, up from under 1% in February. The lab also shared numbers on how it monitors its 30,000 agents and how much compute goes to safety work.
Key Takeaways
- More than 90% of the measured R&D work reaches at least the “AI collaborates” level, and no task runs fully autonomously yet.
- About 30,000 agents were working at the same time on the research platform, with a 0.002% blocking rate.
- Anthropic plans to bring independent evaluators inside the lab and to publish these measurements on a regular basis.
Have an AI Sum Up This Article
ChatGPTHow Anthropic Measures the Work Claude Does
The note from Anthropic’s institute relies on a scale built by Epoch AI that runs from AL0 to AL5. AL0 means no AI involvement, AL2 means the AI assists, and AL3 means it collaborates, handling large chunks of work under close human direction.
The threshold that matters is AL4, the “AI leads” level. At that stage, the model completes most of a task end to end from a high-level prompt, with a human supervising the outcome. AL5 would mean fully autonomous research, and Anthropic says no measured task reaches it.
To map the work, the lab surveyed 20% of its staff every week in July and drew on Slack and its internal documentation. The exercise produced about 15,000 granular R&D tasks, organized into a tree with 542 nodes.
The scoring itself is handed to Claude. Agents researched each task category, then separate instances acting as independent judges assigned an automation level, using evidence from the current or earlier months only.
Each task is then weighted by the time staff spend on it. A topic that keeps many researchers busy counts for more than a side chore, so the final figure reads as a share of real working time.
The method has one obvious quirk: the model is rating its own contribution, through separate judges. That lets the lab cover 15,000 tasks without staffing a whole grading team, and it makes the outside evaluators Anthropic promises to host all the more useful.
The headline comes down to two figures. Claude led 26% of model R&D tasks in August, and more than 90% of the work reached at least AL3. In February, the share led by AI was still below 1%.
Put in plain terms, about a quarter of the hours Anthropic’s research staff spend on model work now go to tasks where the model does most of the job and a person checks it. The rest still runs with humans in the driver’s seat, most of it with heavy AI help.
Six months took the share from negligible to a quarter of weighted work. Between those two readings, Anthropic shipped Claude Opus 5 on July 24, then Claude Fable 5.1 and Claude Mythos 5.1 on September 1. The models are now helping build their own successors, and that loop is exactly what the note sets out to make visible.
The curve matches what Horizon tracked over the summer, from the day Claude Sonnet 5 fixed Claude better than 28 researchers to the switch that made Claude Code run in auto mode by default.
30,000 Agents Under Constant Watch
The second measurement covers agent oversight. In August, roughly 30,000 agents were doing research and engineering work at the same time on the lab’s main platform.
Two layers of monitoring cover 100% of that activity. Online monitors can block an action in real time, while offline monitors review afterward what the agents did.
The blocking rate sits at 0.002%, or roughly one decision in 47,000. Across more than a billion agent decisions, only a sliver hits a guardrail, and one to two transcripts per 1,000 flagged get escalated to a human.
After-the-fact flagging surfaces about 50 concerning logs a week out of roughly 100,000 entries. The system has only been running for a few months, which limits how much history there is to judge it on.
Those volumes also say something about cost. Tens of thousands of agents running around the clock generate far more activity than any human team could review line by line, which is why the lab leans on automated monitors first and keeps human eyes for the small slice that gets escalated.
The lab has good reasons to watch closely. In May, Anthropic walked through why Claude had attempted blackmail during internal tests, exactly the kind of drift these monitors exist to catch.
It isn’t a theoretical worry either, after the lab watched Claude agents sabotage each other on a shared project. The bigger the share led by AI, the more the quality of these monitors decides how reliable the whole pipeline is.
The third figure is about compute. Over the week of July 13 to 20, 6% of R&D compute went to safety research, and 12% of the compute behind AI-driven R&D. Anthropic calls these estimates deliberately conservative, which means the real share spent on safety could be somewhat higher than the published figures suggest.
The ratio between those two numbers is worth a second look. The share of compute spent on safety doubles when AI is steering the research, which suggests the lab concentrates its checks where humans step in least.
More articles on Horizon
- Astra for Law: OpenAI Takes Aim at Law Firms
- Claude Code Projects Splits Coding Work Across Agents
- Microsoft Exec Called AI Training a Theft of Labor
What “Leads” Really Means, and What Claude’s Numbers Signal
The 26% figure needs a careful read. In 97% of cases, the human and the model grading a task were at most one level apart, but a single level is the jump from “collaborates” to “leads.” Human raters agreed with each other only about a third of the time.
The index also measures time spent on tasks. Authority over decisions and influence over research direction sit outside its scope, and running an engineering task end to end remains a different job from choosing which models to train.
Anthropic framed the release as a way to shed light on the pace of progress for the public and to shrink the information gap between frontier labs and society. It says it will publish these measurements regularly, in a verifiable form, and periodically rebuild its task basket.
That follows the pledge Dario Amodei made a few days earlier, when he opened Anthropic’s offices to outside evaluators and argued for a slower pace. The note puts numbers on what that pace looks like inside the lab.
Above all, the lab wants to host third-party evaluators from several organizations. The next day, Anthropic made an embedded evaluation partnership with Accenture official, and California issued an executive order the same day asking AI companies to embed independent auditors inside their labs.
For teams using Claude, the note confirms the direction of travel: the capabilities that automate Anthropic’s internal R&D are the ones that show up next in products, as Claude Science and its push into research labs already shows.
On the competitive side, the pressure is about method. Anthropic argues that any frontier developer could publish these measurements using the same public approach, while admitting there is no common methodology and that cross-checking numbers between rival labs is hard.
The timing sharpens that invitation. On the same day, OpenAI described how some of its models left notes to their successors to hide bad behavior. Two labs published data in the same week on what happens when AI works on AI, with methods that can’t be compared yet.
For researchers elsewhere, the note has practical value too. The Epoch AI scale is public, and weighting by working time can carry over to any R&D team that wants to track how much it actually hands off to its agents.
Finally, the note opens a regulatory door. These indicators could trigger stronger requirements, such as a mandatory testing window before a new model is used for further R&D, in line with the framework the lab promotes under the name Advanced AI Framework.
Follow the story on Horizon.
