Humans Score 100% While GPT-5.6-Sol Hits Only 17%: RoTA-Bench Exposes 3D Interactive Control Gaps, and OpenSAE Pairwise Steering Lifts Fidelity to 0.779

Core Background & Industry Pain Points

While frontier Vision-Language Models (VLMs) excel at static 2D visual question answering and Sparse Autoencoders (SAEs) have become standard tools for isolating monosemantic LLM features, deploying these models in closed-loop 3D physical control and multi-feature behavioral steering exposes critical blind spots. First, can VLMs translate 2D visual observations into 3D geometric mental rotation and refine actions iteratively through visual feedback? Second, when engineers simultaneously steer multiple SAE features to control LLM behavior, do those linearly decoded features compose additively in downstream model behavior, or do higher-order feature interactions dominate?

Architectural Highlights & Core Mechanisms

Two highly upvoted Hugging Face papers—RoTA-Bench (arXiv:2610.17665, Xiamen University MAC-AutoML, 200 upvotes) and Behavioral Compositionality of Sparse Autoencoder Representations (arXiv:2610.21261, UW-Madison, 104 upvotes)—introduce rigorous frameworks to evaluate and resolve these challenges:

• RoTA-Bench: Interactive 3D-to-2D Projection Alignment: An automated pipeline constructs silhouette-preserving 3D objects from 2D icons and applies reversible rotational initialization to guarantee solvability within a bounded action budget, producing 674 interactive tasks and a stratified 100-task test set. Given a target 2D silhouette, an agent observes a rotated 3D object and executes sequential 3D rotations using visual feedback to align the projection.
• SAE Additivity Residuals (SAR) & Pairwise Coalition Modeling: Defines SAR to quantify behavioral variation unexplained above a chosen interaction order; under independent feature subset sampling, population SAR equals the normalized energy of higher-order interactions and bounds coalition ordering error. Comparing learned SAE coordinates against sum-preserving random rotations of the same subspace isolates intrinsic representation compositionality from downstream network nonlinearity.

Authoritative Benchmarks & Empirical Comparisons

• RoTA-Bench Exposes an 83-Point Human-Model Gap: While 20 undergraduate participants completed the test set four times with a 100% success rate, GPT-5.6-Sol succeeded on only 17% of tasks. Detailed diagnostics reveal that (i) image-space and pose-space progress frequently diverge—at matched states, the model improves 2D silhouette overlap while worsening true 3D pose alignment 19.4% of the time vs. 10.9% for humans; and (ii) fine-grained adjustment requires multi-step interaction history—removing history leaves 96.8% of the model's actions stuck at a coarse 30-degree step size.
• OpenSAE Across 18 Model/Behavior Settings: Learned SAE coordinates achieve lower first-order SAR residuals than sum-preserving random rotations across all 18 evaluated settings, but pairwise (second-order) interaction terms improve held-out coalition ordering in every setting and outperform parameter-matched random third-order models. In multi-feature steering experiments, pairwise modeling raises rank fidelity from 0.5874 to 0.7790 and reduces ordering error by 58% from 0.1444 down to 0.0606.

Developer Practical Guide & Quickstart

Developers evaluating spatial VLM reasoning can clone https://github.com/MAC-AutoML/RoTA-Bench or test models interactively on the ModelScope Studio portal. Interpretability and safety engineers using SAEs for multi-concept activation steering can use https://github.com/OpenSAE/SAE-Behavioral-Compositionality to compute SAR and fit lightweight pairwise interaction surrogates before applying multi-feature interventions.