AgentGarten: Decoupling Physics Simulation from Neural Rendering with Adversarial Forcing for Evolving Agents

Background and the Problem

Embodied AI agents require interactive simulated environments to practice and evolve. However, world creation faces a fundamental trade-off: traditional physics engines (Isaac Sim, MuJoCo) guarantee rigid physical dynamics but suffer from a severe visual domain gap with simplistic textures, while generative video models (e.g., Sora, Gen-3) generate photorealistic pixels but suffer from compounding hallucinations, unphysical geometric distortions, and frame-by-frame error accumulation during extended interaction.

Architecture and How It Works

To reconcile physical fidelity with visual realism, MirroS Lab introduces AgentGarten (arXiv:2610.12374, repo: github.com/MirroS-Lab/AgentGarten):

1. Decoupled Simulation & Neural Rendering Architecture: Simulation backends maintain persistent ground-truth states and execute programmatic rules. Crucially, the simulator never ingests generated video frames, severing the error-compounding feedback loop entirely. Structured geometric conditions are exported via standardized APIs to a neural renderer.
2. Adversarial Forcing via Exact Differentiable Replay: The neural renderer adapts a pretrained video foundation model conditioned on geometry. Adversarial Forcing makes history prefilling differentiable through exact replay, ensuring losses from future frames backpropagate into past observation encodings, reinforced by real-world adversarial supervision to prevent flickering.
3. Inter-Generational Playbook Distillation: Agents perceive real-time rendered streams, interact continuously, and distill successful causal policies into structured 'playbooks' inherited and iteratively refined by succeeding generations of agents.

Benchmarks and Measured Results

Comprehensive evaluations on rendering speed and policy evolution demonstrate:

1. Over 30 FPS Real-Time Interaction: Achieves a steady 37.4 FPS at 480x832 resolution on a single NVIDIA H100 GPU, meeting closed-loop latency thresholds for autonomous agents.
2. Emergent Behaviors in 4 Rounds vs. Tens of Millions: In complex multi-agent hide-and-seek tasks, agents discover obstacle-barricading tactics in just 4 rounds and ramp-assisted wall traversal by round 10, contrasting with 25 million and 100 million rounds required by OpenAI's 2019 pure RL baselines.
3. Zero Compounding Geometric Decay: Across 1,000 continuous interaction steps, conventional autoregressive video world models degraded physical coherence by over 82%, whereas AgentGarten preserved 100% physical fidelity without object disappearance or intersection artifacts.

Getting Started for Developers

The AgentGarten codebase, simulator environments, and model weights are publicly available on GitHub (github.com/MirroS-Lab/AgentGarten). Developers building robotics simulations, autonomous game agents, and spatial intelligence models can construct custom worlds via lightweight code backends and render high-fidelity observations at scale, sidestepping the massive compute overhead of pure end-to-end video generators.