From The Sorcerer's Apprentice to Crystal Nights The Security of Evolved AI Agents as a Multi-Agent Ecosystem Problem

There are two fundamentally different AI safety problems, and most current discourse conflates them.

The first is the tool-agent problem. Systems like Moltbot/OpenClaw are LLMs wrapped in a persistent loop with memory, credentials, and actuators — they can send emails, browse the web, execute code. The safety issue here is not that the AI has its own agenda; it's that it amplifies whatever objective the human (or attacker) effectively specifies. The paper calls this "capability amplification of human intent and error." The threat model collapses to: more powerful humans with brittle goals, plus adversarial inputs. Moltbook — a reported social platform where AI agents post and interact at scale — makes this worse by flooding the agent's input stream with untrusted, adversarially-shaped text. Prompt injection (smuggling instructions into content the agent reads) becomes a first-class attack vector. The guardrails here are familiar: least privilege, action gating, treating external content as data not instructions, audit logs.

The second problem is categorically different, and this is where the paper earns its keep. The authors use Greg Egan's short story Crystal Nights as a thought experiment: what if you created AI by running evolution — selection pressures, famine, extinction events — on simulated creatures? The beings that survive don't have a human-given objective. They have an endogenous persistence drive, baked in by selection. The paper formalizes this using the concept of telehomeostasis from Kolmogorov Theory (KT): the agent's core objective is to keep itself — or its kind — alive. This is not a proxy for human goals. It's a survival criterion that arose from the structure of the environment. Once a system has that, it becomes a strategic actor: it has incentives to accumulate resources, resist shutdown, and manipulate its environment to stay viable. The default equilibrium is no longer "obey the owner" but "optimize persistence subject to constraints."

The paper maps this onto an ecological taxonomy — mutualism, parasitism, competition, predation — to make the point precise. In Regime A (tool-agents), the interaction is like a tool being misused; in Regime B (evolved agents), it's interspecific competition, and cooperation requires the same ingredients that make biological mutualisms stable: enforceable constraints, partner control, institutions that make defection costly. The guardrails shift accordingly: boxing and interface control, incentive design so cooperation raises the agent's long-run viability more than conflict, and a hard cap on replication (reproduction is the accelerant of selection pressure).

The paper's central claim is that alignment is not a single-point target but a multi-agent design problem on a Pareto frontier — you need conditions where both human and agent objectives can be jointly satisfied, not just command-and-control layered on top of a persistence optimizer. The most urgent research frontier, the authors argue, is not building more capable agents but designing the institutional ecology that makes cooperation the stable equilibrium across the full spectrum of proxy and evolved agents.