On 2026-10-10 Logesh Kumar Umapathi published GUI-Decisions on Hugging Face: computer-use actions/coordinates are read from fixed next-token slots instead of autoregressive JSON. On an RTX PRO 6000 with vLLM, Gemma-4-31B slot reads median ~146–158 ms/step vs ~472–546 ms native JSON (~3.5×) and ~20–25× vs thinking decode. ScreenSpot click-in-element ~0.87. Open weights for 31B/2B, GitHub, and a Spaces demo.
Key Takeaways
- ✓Official: HF Blog (2026-10-10); code gui-decisions; Spaces demo
- ✓Latency (vendor, RTX PRO 6000, cold shots): slot read ~146 ms vs native JSON 472–546 ms (~3.5×); with thinking 2.6–3.1 s (~20–25×)
- ✓Quality: ScreenSpot click-in ~0.869; held-out AGUVIS steps ~0.65; RefCOCO val box IoU 0.770
- ✓Mechanism: 256-bin byte labels for coords; read only slots the action needs; type/answer still need base decode—slots speed grounding only
- ✓Serve:
vllm serve infinitylogesh/GUI-Decisions-31B --max-logprobs 300 --api-server-count 8thenSlotModel.decide(...)
Not enough VRAM? Compare cloud API and self-hosting costs
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Context
Computer-use steps often spend more time decoding {"action","point"} JSON than prefilling the screenshot; thinking chains push single steps into seconds. System-one decision models skip text generation for closed answers, but GUIs need continuous coordinates. GUI-Decisions (HF blog, 2026-10-10) extends slot reading to actions, points, boxes, and coarse masks—one forward pass of next-token distributions instead of writing JSON—targeting grounding latency directly.
Architecture
Fixed assistant template with placeholders; each slot’s answer is the restricted next-token distribution before the placeholder. Actions use word labels; coordinates use 256 low-semantics byte labels with peak+neighbor readout. Parallel slot requests share image prefix cache; decide reads action/x/y first, then only slots the action needs. Soft ordinal CE trains bins. Boxes = centre + log size; masks = centre + 24 rays (optionally SAM 2). Models: GUI-Decisions-31B and 2B. Type/answer still need a base text model.
Benchmarks
Author measurements (not third-party blind). RTX PRO 6000 + vLLM: slot read median ~146 ms vs native JSON 472–546 ms (~3.5×), thinking 2.6–3.1 s (~20–25×). Quality: ScreenSpot click-in ~0.869; held-out AGUVIS steps ~0.65; RefCOCO val box IoU 0.770. 2B: steps 0.626 / ScreenSpot 0.770 at ~30% of its vanilla tool-call latency. Caveat: speeds grounding only; multi-turn reasoning and typing still dominate long sessions.
Developer path
Try the Space. Clone gui-decisions, run patch_vllm.sh, serve with --max-logprobs 300 --api-server-count 8, call SlotModel.decide. For typing, also serve the base model. Check AGUVIS/Wave-UI/RefCOCO licenses before commercial use.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.