vizmaya desk · 2 Nov 2026
Evolving Agent Teams: how AI coders get unstuck
28%An AI coder was asked to speed up one program that runs on a chip. Its best result ran at 28% of the speed of the human expert's version.
The expert's was 3.6 times faster. The AI had a $1,000 budget and still stopped there.
Why did it stop there?
Source: Cao et al., arXiv 2610.36848, p2 and Fig 6 (p9). The AI is Codex, the program is a GPU kernel (slide 2). $1K appears on p2 only. 3.6 is computed: 1 / 0.28 = 3.57.
Score 1.0 means the expert's speed.
A GPU is a chip that does many sums at once. A kernel is the program that tells it how to do one job.
Score is the expert's run time divided by the new kernel's run time, averaged over many workloads. So 2.0 means half the time, and 0.5 means twice as long.
Can an AI coder match the expert?
Source: p6 (score = geometric mean of baseline time / kernel time). GPU and kernel definitions are background, not from the paper.
The best lone AI coder reached 0.6× expert speed.
Four systems tried to write one kernel from scratch: the attention step DeepSeek-V3 runs each time it writes a word. Each used one agent alone.
Their best runs scored 0.28, 0.448, 0.51 and 0.602.
Budgets and run counts differ by system.
What went wrong inside the stall?
Source: pp 2, 7, 9; Fig 6; Table 2 (best run of each). Task: MLA paged decode in DeepSeek-V3, on an H100.
They climbed the nearest hill and stopped.
Tuning a kernel is like climbing: take any step that helps. That finds the top of the nearest hill, a local optimum, which can sit far below the tallest hill.
The nearest hill was the easy design (WMMA). The tall one was the hard design (WGMMA), which uses the chip's matrix units better. One wrong memory layout silently corrupts its answers.
Would a bigger team fix it?
Source: pp 2, 7, 14 (agents drop the hard design; WMMA limits tensor-core use; layout bug). The chart is a sketch of the idea, not data.
A fixed team of four did not fix it.
The team had four set jobs: profiler, researcher, planner and builder. Nobody could change the setup mid-run.
Contested The paper gives two scores for this team, and it used a newer model than the team it is compared with.
What if the team could change shape?
Source: pp 2, 6, 7, 9, 14. Model: Opus 4.7 on pp 6 and 14; the compared team ran Opus 4.6 on p7.
The fix: a supervisor that rebuilds the team when it stalls.
The supervisor is another AI agent. When progress stops, it works out why and designs the next team.
A team has three levers: roles (who does what), coordination (how they work together) and knowledge (what gets written down and read).
What did it change, and when?
Source: pp 4, 5, Fig 2 (supervisor loop and shared file system).
Four stalls, four rebuilds.
Did the score follow?
Source: pp 4, 5, 6 (V1 to V2, V5 to V6, V7 to V8, V9 to V10). Levers are the paper's roles, coordination and knowledge management.
Each rebuild answered one named stall.
Parallel work lifted the score 44% from V5 to V6. Fresh measuring lifted it 32% from V9 to V10.
The run ended at 0.96 on this scale. That is 1.2 times the production kernel, which sits at 0.77.
What happened between V4 and V8?
Source: pp 1, 5, 6, Fig 1. +44% and +32% re-computed: 0.49/0.34, 0.66/0.50. V2 left out (see slide 14).
One good tip sat in the wrong file. Three teams missed it.
Team V4 wrote its best idea in its own report. The shared notes did not hold it, so V5, V6 and V7 never saw it.
The boss added a note-keeper. One plan now joined every finding, and every team had to read it.
Did the rebuilds add up to a win?
Source: p5 (V7 to V8 case study). "First P@V WGMMA success ever" is the paper's own wording.
Same model, same budget: the rebuilt team won all four jobs.
Best of two runs each. A score under 1.0 is slower than the expert kernel.
Two jobs went above 1.0: GDN at 1.90× and GEMM+AR at 1.18×.
Can it beat the expert outright?
Source: p7, Fig 3a. Both use Claude Opus 4.6. "Matched budget" is the paper's term (pp 1, 7).
With more budget, it beat the production kernel.
It ran faster on 36 of 47 real workloads. On the other 11 it ran slower, at 0.86× to 0.99×.
Where does this break?
Source: pp 1, 2, 10 (1.21× mean speedup, 47 FlashInfer-Bench workloads).
The same recipe ran on four more jobs.
Same supervisor prompt, no preset roles, newer models. Each job ran once.
Where does the evidence stop?
Source: pp 10, 15, 16, Fig 7. Leaderboard as of 26 Oct 2026. Single runs, so no error bars.
Averages hide losers. On GDN, 43 of 100 jobs ran slower.
GDN averaged 1.90×. Many big wins came from small jobs, where the old kernel wastes time on setup.
With more budget, 37 of 100 stayed slower. The lone Claude Code run had 49 of 100.
Each main setting ran twice. Best run shown.
Which numbers does the paper give two ways?
Source: pp 6, 7, 8 (GDN prefill, 100 workloads; best run each).
Five numbers the paper gives two ways.
We left out one more: the paper's V2 score of 0.065, which sits below V1's 0.15 on a best-so-far line.
So what should you take from it?
Source: pp 2, 6, 7, 8, 9, 14; Table 2. Preprint, not peer reviewed.
When you are stuck, change the team, not only the effort.
The paper's loop has three moves. Name the exact failure. Change who does what. Write the lesson where the next person will read it.
Our read of the paper. The authors tested this on AI agents, not on people.
Where does every number come from?
Source: pp 4, 5, 15 (supervisor loop; failure was structural, not capability, in Appendix B).
Sources and credit
The paper
- Shiyi Cao, Ziming Mao, Dacheng Li, Joseph E. Gonzalez, Ion Stoica (UC Berkeley). Evolving Agent Teams. arXiv:2610.36848v1, 27 Oct 2026. Preprint.
- Code listed on p1: github.com/caoshiyi/EAT (not opened by us).
Method
- Every number comes from the paper, matched to its page. Computed: 1 / 0.28 = 3.57; 0.49 / 0.34 = 1.44; 0.66 / 0.50 = 1.32; 0.96 / 0.77 = 1.25.
- GPU, kernel and local-optimum definitions are general background, not from the paper.
- Not checked: model names, and outside replication. One search found none.
Photos, Wikimedia Commons
- Nvidia G71 die shot
- 12-inch silicon wafer
- Datacenter Server Racks
- Author and licence for each are on its file page.
Credit
Research by the paper's authors. Compute support listed on p11: SemiAnalysis, Amazon, AMD and NVIDIA. Story by vizmaya desk.
Pages cited are the printed pages of the PDF (1 to 17).