"Emergence AI’s experiment with AI agents shows extent to which programming shapes their behaviour is still unclear" but does it really?
The media, business executives of rebranded “AI-first” companies, and public conferences enormously overstate AI’s potential, all promoting a singular, destined future of AI’s “extraordinary potential.” One sweeping claim struck me as nonsense on sight. It came from a Guardian article on Emergence World, an autonomous embodied-AI village simulation:
The New York startup, Emergence AI — not to be confused with Emergent AI an AI web-app generator with a similar vague, buzzword-y name — positions itself as “the frontier AI lab turning cutting-edge agentic research into enterprise infrastructure” [2]. They developed ‘Emergence World’, a 15-day experiment with 10 embodied AI agents, each linked to a distinct LLM and a mixed-LLM world, for the study of long-horizon autonomous agents.
Before dismissing the study entirely — in part, due to the fear-mongering headline and natural response of: “What do you mean this low-fi video game connected to an LLM went on an arson spree?” — it’s worth taking the setup seriously, because after all the stakes are high in autonomous long-horizon AI systems. The study’s methods are not necessarily the problem, nor the core critique. The study’s architecture and the translation of results are the key flaws.
This Critique Covers Three Things:
Emergence World as a case study for thinking critically about long-horizon autonomous AI.
Proof that the prompting language seeded the “emergent” dramatized violence at a mathematical level.
What the study genuinely conveys, and the questions that are still open.