Real-world embodied tasks require agents to operate physical tools under geometric constraints while tracking object state transitions. Despite strong perception benchmarks, current multimodal video LLMs struggle with tool-centric physical reasoning. NTU S-Lab and collaborators introduce EgoTools, the first comprehensive suite for egocentric tool-use reasoning. It features EgoTools-Data (100 hours of synchronized egocentric tool videos with 3D annotations, audio, and dense causal narrations) and EgoTools-Bench (1,000 QA pairs across perception, geometry, procedural progress, and causal inference). Probing reveals frontier models struggle with physical grounding (Gemini-3.1-Pro scores only 51.7% on grounding), while supervised fine-tuning on EgoTools-Data lifts Qwen3-VL-8B-Instruct accuracy from 50.0% to 60.9%.
Key Takeaways
- ✓Pioneers EgoTools, the first egocentric tool-use suite with 100 hours of 3D-annotated reasoning videos
- ✓Unmasks physical grounding gaps in commercial models: Gemini-3.1-Pro scores only 51.7% on tool grounding
- ✓Supervised fine-tuning lifts Qwen3-VL-8B-Instruct accuracy from 50.0% to 60.9% under strict video separation
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the Problem
Embodied operations mediated by tools (handheld assembly, surgical manipulation, tool craft) are central to physical human activities. Current multimodal video models excel at coarse scene descriptions but falter on fine-grained tool-centric reasoning: understanding Hand-Tool-Object spatial geometry, physical affordances, workflow milestones, and causal deformative impacts on target objects.
Architecture and How It Works
NTU S-Lab and collaborators introduce EgoTools, the first holistic framework for physical tool understanding:
- EgoTools-Data Corpus: 100 hours of egocentric tool manipulation video featuring synchronized audio, dense temporal timestamps, supplementary 3D spatial captures, and reasoning-heavy narrations encoding physical intentions.
- EgoTools-Bench Diagnostic Benchmark: 1,000 rigorous QA challenges split into four diagnostic tracks: Perception & Grounding, Hand-Tool Geometry, Procedural Progress, and Causal Reasoning.
- Source-Isolated Evaluation: Guarantees zero test-time data contamination via strict video-level partition protocols.
Benchmarks and Measured Results
Evaluated on frontier visual foundation models:
- Physical Grounding Bottlenecks in Frontier Models: While Gemini-3.1-Pro achieves 66.9% overall accuracy, it plummets to 51.7% on the Perception & Grounding track, highlighting deep deficits in physical visual evidence binding.
- SFT Drives a 10.9% Leap: Supervised fine-tuning on EgoTools-Data lifts Qwen3-VL-8B-Instruct from 50.0% to 60.9% (+10.9 percentage points) under rigorous video-level separation.
- Zero-Shot Affordance Transfer: Fine-tuned representations successfully predict affordance contact regions and action directions on entirely novel industrial tools.
Getting Started for Developers
EgoTools data, benchmarks, and model checkpoints are available on GitHub. Developers creating AR assistant plugins for smart glasses (e.g., Vision Pro, Meta Ray-Ban) or autonomous robotic manipulators can utilize EgoTools to teach vision models how to ground physical tool interactions in the real world.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.