Nobody knows yet how to organize a team of AI agents
Splitting work across my crew of coding agents seemed to lower the quality, and the few studies so far leave the best way to organize a team of agents open.
For the past few weeks I've run most of my work through Firstmate. I talk to one agent, the first mate, and it runs a crew of coding agents for me. It's very effective, and for bigger fleets you can add second mates, which are supervisors with crews of their own. But whenever I split a job across agents, the quality seemed to go down. It was a feeling, and I couldn't put my finger on why. Then I read a Google DeepMind essay on the economics of AGI. It argues that the big economic change hinges on agents learning to coordinate, and it says "Today's AI systems coordinate poorly in practice". The evidence it links for that sentence is a coding benchmark, and that got me reading. So what is the best way to organize a team of agents?
The benchmark is CooperBench, and it puts a number on my feeling: two agents that each build one feature of the same repository succeed much less often than one agent that builds both, even though the two can message each other. A preprint on teams of coding agents did find teams that beat one agent. The cheap version that works keeps the plan and all of the code with one strong lead, and its cheap explorers, which the paper calls planners, only report what they find. In that paper, the split that scored best also changed with the benchmark. I couldn't find convincing evidence for any one way to organize a team of agents, which really surprises me. Until someone finds one, I keep tightly coupled work in one agent and give cheap helpers bounded jobs that only bring back information, like searching, reading or running a command.