USC, UCLA, and CMU Release Grid-Bench for Multi-Step Spatial Reasoning: 8,928 Executable Instances Reveal Severe State Drift Across 64-Step Trajectories

Background and the Problem

As LLM-based coding agents and simulation frameworks take on spatial computing tasks—such as procedural CAD generation, grid-based graph processing, and game state forecasting—they must maintain consistent spatial world representations across multi-step instructions (reflecting regions, moving components, aggregating neighborhoods). However, existing vision-language and reasoning benchmarks evaluate static question-answering rather than tracking how internal representations evolve over long action sequences. Researchers from USC, UCLA, and CMU introduced Grid-Bench (arXiv:2610.22312, GitHub: grid-bench/Grid-Bench), an executable spatial benchmark spanning 2D, 3D, and 4D discrete manifolds.

Architecture and How It Works

1. Eight Grounded Spatial Rule Families Across Multi-Dimensional Manifolds:

• Defines 8 mathematically rigorous spatial transformation rules: PAINT, REFLECT, STENCIL (modular neighborhood aggregation), MIX, RELABEL (connected-component coloring), LAYER, MOVE, and COMPRESS (tile sliding);
• Tests across 2D (16×16), 3D (6×6×7), and 4D (4×4×4×4) grids (252–256 cells) across 6 history lengths ($T = 0, 1, 8, 16, 32, 64$), generating 8,928 independently executable evaluation instances.

1. Deterministic Python Execution & Diagnostic Matrix:

• Because reference execution provides an exact ground truth, performance is broken down into complete-state exact matches, cell-wise accuracy on modified regions (C-Acc), and cell-wise accuracy on static regions (U-Acc).

Benchmarks and Measured Results

Benchmarked across six frontier models (GPT-5.6 Sol, Claude 5 Sonnet, GPT-5.4 Mini, GLM-5, Kimi K2.5, MiniMax M2.5) with 128k output tokens and no external tools (Tables 10, 11, 13, 15):

• Short Trajectories ($T=1$): GPT-5.6 Sol yields 183/192 exact world matches, MiniMax M2.5 reaches 150, Kimi K2.5 hits 148, and Claude 5 Sonnet scores 137;
• Intermediate History Divergence ($T=16$):
• GPT-5.6 Sol maintains 98.0% cell accuracy (87/192 exact worlds), and Claude 5 Sonnet posts 96.8% cell accuracy (61/192 exact worlds);
• Mid-tier and open models suffer sharp drops: GPT-5.4 Mini falls to 5 exact matches (82.4% cell acc), GLM-5 to 8 exact (80.9%), Kimi K2.5 to 18 exact (80.6%), and MiniMax M2.5 to 11 exact (64.3%);
• Long-Horizon Breakdown ($T=64$):
• Over 64 consecutive transitions, mid-tier models completely break down: GLM-5 and Kimi K2.5 hit 0/192 exact matches, while GPT-5.4 Mini and MiniMax M2.5 recover just 1;
• Frontier reasoning flagships retain non-trivial tracking: GPT-5.6 Sol achieves 25 exact world recoveries (81.6% cell acc) and Claude 5 Sonnet yields 14 exact recoveries (90.5% cell acc);
• Total Exact Matches across $T=8..64$ (Table 11, /768): GPT-5.6 Sol dominates with 314 exact matches, Claude 5 Sonnet achieves 201, outstripping Kimi K2.5 (57), MiniMax M2.5 (48), and GLM-5 (28).

Getting Started for Developers

• Open-Source Benchmark: The evaluation harness and datasets are accessible at https://github.com/grid-bench/Grid-Bench.
• Architectural Implications: Findings demonstrate that even 100B+ parameter LLMs fail to track spatial discrete dynamics past 16 steps without external aids. When engineering Coding Agents for CAD refactoring or discrete state machines, developers should never rely on the model's internal scratchpad memory; instead, anchor execution to an external deterministic state engine that executes model-proposed atomic commands.