BrickBench: Stanford and Inria Unveil Benchmark for Agentic 3D LEGO Assembly, Exposing Gaps Between Coding Agents and Humans

Background and the Problem

While foundation models excel at writing traditional software, extending coding agents to discrete physical engineering and mechanical assembly remains an open frontier. Contemporary 3D generative architectures typically output continuous, non-manufacturable surface meshes. In contrast, real-world physical design requires selecting discrete modular components from standardized inventories while simultaneously satisfying micro-level interlocking mechanics and macro-level structural stability under gravity. Evaluating whether autonomous coding agents can reason over complex combinatorial constraints in 3D space requires rigorous physics-grounded benchmarks.

Architecture and How It Works

To evaluate spatial and physical reasoning in coding agents, Stanford University and Inria present BrickBench and the BrickAgent harness (arXiv:2610.12452, brickben.ch):

1. BrickAgent Execution Sandbox: Provides an interactive Python environment where coding agents construct assemblies, query parts databases, execute physics-based collision tests, and inspect intermediate structures.
2. Tri-Partite Evaluation Metric Suite:

• Physical Validity: Assesses geometric collision-free placement, valid stud-tube connectivity, and static equilibrium under gravitational loading.
• Semantic Alignment: Measures text-to-structure alignment using multi-view VLM inspection and semantic feature matching.
• Design Quality: Evaluates part economy, symmetry, structural redundancy, and aesthetic cohesion.

1. Scaled Part Regimes: Tests agents across unconstrained libraries, inventory-limited settings, and adversarial component distributions to assess resource-aware planning.

Benchmarks and Measured Results

Benchmarked across leading coding agents including Claude 3.5 Sonnet and GPT-4o:

1. Physical Validity vs. Human Engineering Gap: Frontier agents achieve over 70% validity on basic assembly tasks, successfully avoiding intersecting meshes. However, on complex multi-tier assemblies, agents frequently generate mechanically unstable structures that collapse under gravity.
2. Over 45% Part Redundancy Over Human Designs: Coding agents exhibit high inefficiency, consuming 45% more bricks than human builders to achieve equivalent geometric coverage due to lack of global structural intuition.
3. +28.4 Points via Closed-Loop Refinement: Granting agents three turns of simulation feedback and code revision within the BrickAgent sandbox elevates structural validity by 28.4 percentage points, highlighting the power of execution feedback in physical modeling.

Getting Started for Developers

The BrickBench environment is accessible at brickben.ch. Engineering teams building CAD/CAM automation, robotic manufacturing pipelines, or procedural 3D systems should treat physical assembly as a structured coding task. Pair coding agents with deterministic physics checkers and collision validators, enforcing procedural generation scripts that guarantee manufacturable stability prior to downstream fabrication.