Transformers Stop Thinking Too Early: A Single Early-Layer Rank-8 LoRA Unlocks Deep Reasoning Relay from 15.5% to 99%

核心背景与行业痛点

While context lengths have expanded to millions of tokens, pretrained transformers fail to use their architectural depth for complex multi-hop reasoning. Systematic probing reveals that thirteen leading base models (spanning Qwen and Llama families) reliably follow references for only 1.4 to 3.6 lines before freezing computation across remaining layers. Simply increasing recurrent loops provides no remedy.

架构亮点与底层机制

Researchers from Tsinghua and Peking University diagnose this early computational stalling and propose a targeted mechanistic solution:

1. Single-Layer Rank-8 LoRA Insertion: Inserts a minuscule rank-8 LoRA adapter into a single early layer while freezing 100% of the base model weights.
2. Initiating a Computational Relay: The LoRA injects compact chain identifiers into the residual stream, triggering a layer-by-layer relay through middle layers.
3. Activating Frozen Attention Heads: Downstream frozen attention heads utilize these identifiers to trace progressively further up the dependency graph.
4. Analytical Layer Localization: A frozen-model probing diagnostic accurately predicts the optimal intervention layer across held-out architectures without trial-and-error fine-tuning.

权威 Benchmark 与实测跑分对比

Evaluated on synthetic pointer chain tracking and the real-world MuSiQue multi-hop benchmark:

1. Qwen3-8B Surges from 15.5% to 99%: On 24-line reference chains, unassisted Qwen3-8B collapses at 15.5%, whereas the single-layer LoRA achieves 99.0% exact accuracy, scaling reliably beyond 50 steps with further training.
2. 160+ Lines on Looped Models: Ouro-1.4B scales from 60 lines after four loops to over 160 continuous reference lines after eight loops.
3. Multi-Hop Reasoning Improvements: Outperforms frozen baselines on the MuSiQue multi-hop reading comprehension benchmark, confirming real-world efficacy.

开发者实战落地与开箱指南

Code, probing suites, and interactive demos are open-sourced on GitHub. Engineering teams deploying coding agents or complex contractual analysis pipelines can deploy single-layer micro-LoRAs to unlock the dormant computational depth of open-weight foundation models at negligible training cost.