R
Link preview
New LLM Coordination Benchmark - Benchmarking Open-Ended Multi-Agent Coordination in Language Agents [R]
Can LLM agents coordinate in long-horizon, open-ended worlds? We evaluate 13 modern LLMs in a new benchmark where agents must work together to explore, communicate, trade resources, craft tools, build structures, and fight mobs. TL;DR: Most agents struggle, averaging only ~6% normalised return. Yet on the hardest setting, zero-shot Gemini 3.1 Pro performs comparably to the best MARL agent trained for 1 billion environment steps. More broadly, we find coordination is a distinct bottleneck beyond long-horizon task competence, with communication having the largest effect in our harness ablations. Paper: https://arxiv.org/abs/2606.08340 Project page and leaderboard: https://alem-world.github.io Code: https://github.com/alem-world/alem-env Interactive traces: https://alem-world.github.io/traces.html Feel free to ask any questions! submitted by /u/ktessera [link] [Kommentare] reddit.com · reddit.com ↗
Can LLM agents coordinate in long-horizon, open-ended worlds? We evaluate 13 modern LLMs in a new benchmark where agents must work together to explore, communicate, trade resources, craft tools, build structures, and fight mobs. TL;DR: Most agents struggle, averaging only ~6% normalised return. Yet on the hardest setting, zero-shot Gemini 3.1 Pro performs comparably to the best MARL agent trained for 1 billion environment steps. More broadly, we find coordination is a distinct bottleneck beyond long-horizon task competence, with communication having the largest effect in our harness ablations. Paper: https://arxiv.org/abs/2606.08340 Project page and leaderboard: https://alem-world.github.io Code: https://github.com/alem-world/alem-env Interactive traces: https://alem-world.github.io/traces.html Feel free to ask any questions! submitted by /u/ktessera [link] [Kommentare]
Comments