A test of agentic workflows for software engineering

Setup

Nine agent workflows were given the same prompt: to implement a detailed written specification in an existing Python codebase.

Specifically, the task was to implement complex structured logging wrapped around a small app with strict accounting requirements. On average, agents wrote ~2,250 new lines of Python code, spanning ~11 new files and 17 files touched in total.

Success was judged by:

• Correctness: linter, formatter, type-checking, and tests all pass, and code was reviewed for hidden bugs.
• Consistency with the spec.
• Hackiness: bypassing linting/type-checking with noqa labels or implementing "fragile" solutions.
• Code readability and maintainability.
• Bonus points for completeness or "future readiness" when the code anticipates future work elegantly.

Evaluations were performed with LLM-as-a-judge using Opus 5.5 and Astra 6 independently, and then averaged.

Harness:

• OpenAI models ran in Codex.
• Anthropic models ran in Claude Code.
• Cursor/SpaceXAI models ran in Cursor.

Costs were calculated at published API rates based on the exact breakdown of token types and amounts recorded in the logs. See the detailed cost breakdown addendum below.

Limitations

• No Fable: special data retention rules are a non-starter.
• No Chinese models: often banned entirely in highly regulated environments.
• Meta Muse Spark 1.3 not tested
• Gemini not tested
• All models ran on "medium" reasoning. See point #6 below.
• Only one trial was run for each workflow.

Results

My Learnings

1. Opus 5.5 actually wrote the best code.
   By every standard of real engineering quality, it was the best. In this experiment, it lost on a technicality: it failed to ensure its linter was passing at the end; namely, it was missing docstrings on test cases. This is a quick fix with prompting, not a limitation of the model. If you are doing foundational engineering work, you want to rely on Opus. Even though it is more expensive and much slower than other "more intelligent" models, it seems to have some secret sauce which makes its work really good.
   Fable (if you allow it): I'm guessing it would dominate all the others.
2. Arguably the main takeaway here is "just use Astra".
   Astra is outstanding. It's ridiculously fast and it's ridiculously token efficient and its quality is excellent. You get frontier intelligence, at speed, and at a very good price. And it doesn't need a lot of special prompting to just get it right.
3. Composer-2.5 is massively underappreciated.
   It is a star. It is the only fast, cheap model tested that can actually code with decent speed and quality. It is competing admirably in this benchmark with GPT-6-Astra with just a little code review help from Grok. In this combination, it was significantly faster than Astra, at well under half the price, and at comparable quality.
   Chinese models, if you allow them, I expect will show a similar value prop.
4. Code reviews are cheap, even with very expensive models.
   They don't have long agentic loops racking up millions of cache read tokens. You should always throw the most intelligent models you have at code reviews for a huge ROI.
5. "Cheap" models are not "good enough".
   Haiku, Luna, and Sonnet (not pictured) are poor coders. Do not use them. Despite promising benchmarks, in reality they give worse quality, a longer run time, and a higher cost than the "expensive" models.
6. Try bumping up reasoning effort.
   Sol was shockingly fast and cheap with medium reasoning. It gave a minimal solution compared to other models, with decent but not great quality. Reasoning is a small portion of the cost. This experiment should be rerun cranking up reasoning to high, xhigh, max. My hypothesis is that cost and time increases will be modest and quality will be much higher, which may turn the tables on these rankings.
7. Try using an expensive model for a quick first draft.
   Cost is driven by cache reads during long chains of tool calls, e.g. fixing code in a loop against tests/linters. This experiment should be rerun with a more intelligent model instructed to draft an initial solution without using linters or running tests. Hypothesis: this will give a very strong, intelligently engineered foundation, and then much weaker models could run the long, expensive agentic iterations to pass style and test gates.

Addendum: detailed cost breakdown

Reference pricing