4DCodeBench: Stanford and MIT Benchmark Coding Agents on Inverse Graphics of Dynamic Physical Scenes

核心背景与行业痛点

Inverse graphics aims to reconstruct the underlying geometry, appearance, and physical mechanics of the physical world from raw sensory observations, translating visual inputs into symbolic, executable programs. While contemporary coding agents demonstrate strong performance on competitive programming and static 3D mesh synthesis, real-world manipulation requires understanding dynamic 4D spatiotemporal reality (3D space + 1D time). Autonomous robots manipulating non-rigid textiles or predicting fluid dynamics require models that perceive visual kinematics and formulate corresponding physical simulation code. Until now, the AI community lacked a rigorous benchmark for evaluating code-driven 4D inverse graphics of continuum dynamics.

架构亮点与底层机制

Researchers from Stanford University, MIT, and Johns Hopkins introduce 4DCodeBench (arXiv:2610.03715), a holistic evaluation platform:

1. Continuum Physics Phenomena Coverage: Incorporates both real-world video recordings and ground-truth numerical simulations, spanning complex non-rigid deformation (elastic collisions), fluid dynamics (liquid pouring and splashing), and structural fractures.
2. Synthesis of Executable Graphics & Simulation Code: Evaluated agents must analyze video streams to produce compact, executable simulation programs using numerical graphics frameworks (e.g., Taichi, differentiable physics engines) that simulate and render the full dynamic sequence.
3. Dual Spatiotemporal and Physical Evaluation: Assesses reconstructed simulations via both visual rendering fidelity metrics (PSNR, SSIM, temporal LPIPS) and intrinsic physical kinematics (center-of-mass trajectories, velocity fields, momentum conservation), penalizing visually coherent hallucinations that violate physical mechanics.

权威 Benchmark 与实测跑分对比

Benchmarked across frontier foundation models including GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro:

1. Disconnect Between Static 3D and Dynamic 4D Capabilities: Models that achieve over 80% geometric accuracy on static 3D tasks suffer catastrophic collapse on 4D continuum dynamics, with physically valid execution rates plummeting below 15%.
2. Pervasive Physics Hallucinations: Agents frequently resort to hardcoded geometric keyframing rather than synthesizing differential equations of motion, failing to balance material viscosity or stress tensors when encountering sudden collisions.
3. Significance of Algorithmic Abstractions: Providing agents with structured numerical physics abstractions substantially boosts success rates, indicating that domain-specific physics scaffolding is essential for spatial world modeling.

开发者实战落地与开箱指南

4DCodeBench codebase, dataset assets, and evaluation harnesses are available on GitHub (4DCodeBench/4DCodeBench). Machine learning teams architecting physical world models, robotics simulation environments, and generative game development agents can leverage 4DCodeBench to evaluate models beyond surface text, charting the progression of AI agents that understand physical dynamics through code.