Close-up photo of the silicon die of an older NVIDIA GPU chip
Photo: Nvidia G71 die shot, Wikimedia Commons. An older NVIDIA GPU, not the chip tested in the paper. Author and licence on the file page.

vizmaya desk · 2 Nov 2026

Evolving Agent Teams: how AI coders get unstuck

28%An AI coder was asked to speed up one program that runs on a chip. Its best result ran at 28% of the speed of the human expert's version.

The expert's was 3.6 times faster. The AI had a $1,000 budget and still stopped there.

Why did it stop there?

Source: Cao et al., arXiv 2610.36848, p2 and Fig 6 (p9). The AI is Codex, the program is a GPU kernel (slide 2). $1K appears on p2 only. 3.6 is computed: 1 / 0.28 = 3.57.

Score 1.0 means the expert's speed.

A GPU is a chip that does many sums at once. A kernel is the program that tells it how to do one job.

Score is the expert's run time divided by the new kernel's run time, averaged over many workloads. So 2.0 means half the time, and 0.5 means twice as long.

A 12-inch silicon wafer
Photo: 12-inch silicon wafer, Wikimedia Commons. Chips are cut from wafers like this. Author and licence on the file page.

Can an AI coder match the expert?

Source: p6 (score = geometric mean of baseline time / kernel time). GPU and kernel definitions are background, not from the paper.

The best lone AI coder reached 0.6× expert speed.

Four systems tried to write one kernel from scratch: the attention step DeepSeek-V3 runs each time it writes a word. Each used one agent alone.

Their best runs scored 0.28, 0.448, 0.51 and 0.602.

Budgets and run counts differ by system.

What went wrong inside the stall?

Source: pp 2, 7, 9; Fig 6; Table 2 (best run of each). Task: MLA paged decode in DeepSeek-V3, on an H100.

They climbed the nearest hill and stopped.

Tuning a kernel is like climbing: take any step that helps. That finds the top of the nearest hill, a local optimum, which can sit far below the tallest hill.

The nearest hill was the easy design (WMMA). The tall one was the hard design (WGMMA), which uses the chip's matrix units better. One wrong memory layout silently corrupts its answers.

Would a bigger team fix it?

Source: pp 2, 7, 14 (agents drop the hard design; WMMA limits tensor-core use; layout bug). The chart is a sketch of the idea, not data.

A fixed team of four did not fix it.

The team had four set jobs: profiler, researcher, planner and builder. Nobody could change the setup mid-run.

Contested The paper gives two scores for this team, and it used a newer model than the team it is compared with.

55%Page 2, introduction
0.31×Page 9, Fig 6

What if the team could change shape?

Source: pp 2, 6, 7, 9, 14. Model: Opus 4.7 on pp 6 and 14; the compared team ran Opus 4.6 on p7.

The fix: a supervisor that rebuilds the team when it stalls.

The supervisor is another AI agent. When progress stops, it works out why and designs the next team.

A team has three levers: roles (who does what), coordination (how they work together) and knowledge (what gets written down and read).

What did it change, and when?

Source: pp 4, 5, Fig 2 (supervisor loop and shared file system).

Four stalls, four rebuilds.

The stall
What the supervisor changed
Levers
Drift: the agent slid from the hard design to plain code, with no profiling.
Added a rule enforcer: no easy-path commits, one change per commit. Started a shared notes file.
roles, coordination, knowledge
Dead end: three teams hit the same bug. One session was too short to debug and build.
Ran a debugger and an optimizer side by side on separate branches.
roles, coordination
Lost tip: a key idea sat in one team's report, not in the shared notes.
Added a note-keeper who writes one plan from all findings. Every team must read it.
roles, knowledge
Stale numbers: a new design made all old measurements useless.
Started a team that measures first, then optimizes.
roles, coordination

Did the score follow?

Source: pp 4, 5, 6 (V1 to V2, V5 to V6, V7 to V8, V9 to V10). Levers are the paper's roles, coordination and knowledge management.

Each rebuild answered one named stall.

Parallel work lifted the score 44% from V5 to V6. Fresh measuring lifted it 32% from V9 to V10.

The run ended at 0.96 on this scale. That is 1.2 times the production kernel, which sits at 0.77.

What happened between V4 and V8?

Source: pp 1, 5, 6, Fig 1. +44% and +32% re-computed: 0.49/0.34, 0.66/0.50. V2 left out (see slide 14).

One good tip sat in the wrong file. Three teams missed it.

Team V4 wrote its best idea in its own report. The shared notes did not hold it, so V5, V6 and V7 never saw it.

The boss added a note-keeper. One plan now joined every finding, and every team had to read it.

Did the rebuilds add up to a win?

Source: p5 (V7 to V8 case study). "First P@V WGMMA success ever" is the paper's own wording.

Same model, same budget: the rebuilt team won all four jobs.

Best of two runs each. A score under 1.0 is slower than the expert kernel.

Two jobs went above 1.0: GDN at 1.90× and GEMM+AR at 1.18×.

Can it beat the expert outright?

Source: p7, Fig 3a. Both use Claude Opus 4.6. "Matched budget" is the paper's term (pp 1, 7).

1.21×

With more budget, it beat the production kernel.

It ran faster on 36 of 47 real workloads. On the other 11 it ran slower, at 0.86× to 0.99×.

Where does this break?

Source: pp 1, 2, 10 (1.21× mean speedup, 47 FlashInfer-Bench workloads).

The same recipe ran on four more jobs.

Same supervisor prompt, no preset roles, newer models. Each job ran once.

Setting
Result
Read with care
KDA prefill, AMD MI350X
3.01× vs vLLM's hand-written kernels
Started from a port of them, scored 0.99.
KDA prefill, NVIDIA B200
3.44× vs FlashKDA
Started from a port, scored 1.96×.
Model serving on a Mac (M4 Pro)
3.32× vs the naive engine
A serving engine, not a GPU kernel.
RoboDojo, 42 robot tasks
62.1 vs 55.3, best leaderboard entry
The policy could call a vision-language-action model and an LLM.

Where does the evidence stop?

Source: pp 10, 15, 16, Fig 7. Leaderboard as of 26 Oct 2026. Single runs, so no error bars.

Averages hide losers. On GDN, 43 of 100 jobs ran slower.

GDN averaged 1.90×. Many big wins came from small jobs, where the old kernel wastes time on setup.

With more budget, 37 of 100 stayed slower. The lone Claude Code run had 49 of 100.

Each main setting ran twice. Best run shown.

Which numbers does the paper give two ways?

Source: pp 6, 7, 8 (GDN prefill, 100 workloads; best run each).

Five numbers the paper gives two ways.

What
One place
Another place
Fixed four-role team
55% p2
0.31× p9
Model behind that team
Opus 4.7, pp 6, 14
"same model", p9
Rebuilt team, MLA job
0.74× pp 2, 7, 9
0.75× p7; 0.721×, 0.685× Table 2
GEMM+AR, extra budget
1.49× p8, section 4.2
1.56× p8, section 4.3
Best earlier system
0.61× p2
0.602×, 0.598× Table 2

We left out one more: the paper's V2 score of 0.065, which sits below V1's 0.15 on a best-so-far line.

So what should you take from it?

Source: pp 2, 6, 7, 8, 9, 14; Table 2. Preprint, not peer reviewed.

When you are stuck, change the team, not only the effort.

The paper's loop has three moves. Name the exact failure. Change who does what. Write the lesson where the next person will read it.

Our read of the paper. The authors tested this on AI agents, not on people.

Rows of server racks in a data center
Photo: Datacenter Server Racks, Wikimedia Commons. Author and licence on the file page.

Where does every number come from?

Source: pp 4, 5, 15 (supervisor loop; failure was structural, not capability, in Appendix B).

Sources and credit

The paper

Method

  • Every number comes from the paper, matched to its page. Computed: 1 / 0.28 = 3.57; 0.49 / 0.34 = 1.44; 0.66 / 0.50 = 1.32; 0.96 / 0.77 = 1.25.
  • GPU, kernel and local-optimum definitions are general background, not from the paper.
  • Not checked: model names, and outside replication. One search found none.

Photos, Wikimedia Commons

Credit

Research by the paper's authors. Compute support listed on p11: SemiAnalysis, Amazon, AMD and NVIDIA. Story by vizmaya desk.

Pages cited are the printed pages of the PDF (1 to 17).