REMORY: HKUST and Alibaba Introduce Residual Memory for Agent Context Compaction, Matching Full Context with Only 5.2% Token Overhead

Background and the Problem

In operating system automation, autonomous web navigation, and enterprise DevOps workflows, agents regularly execute dozens or hundreds of sequential tool calls. As interaction logs accumulate into hundreds of thousands of tokens, teams face prohibitive API costs alongside severe lost-in-the-middle context dilution. Current compaction approaches rely on textual summarization or naive token pruning. However, natural language summaries are lossy by definition, stripping away minute environmental details such as port bindings, UUIDs, environment variables, and stack traces. When agents require these details later in long horizons, the lost precision triggers redundant execution loops and catastrophic hallucinations.

Architecture and How It Works

To resolve this tension between memory retention and inference efficiency, HKUST and Alibaba propose REMORY (arXiv:2610.11287), introducing residual connections to context compaction:

1. Residual Memory Architecture in Sequence Space: Drawing inspiration from ResNet skip connections, REMORY decouples memory management into two parallel paths. The trunk path provides a compact, human-readable natural language summary to maintain global narrative flow. The residual path introduces a bounded sequence of continuous soft memory tokens trained to reconstruct missing low-level execution artifacts, tool outputs, and state transitions.
2. Bi-Directional Multi-Task Reconstruction Alignment: During training, a lightweight memory encoder maps interaction histories into soft tokens, supervised by contrastive learning and multi-task reconstruction objectives that faithfully encode granular parameters.
3. Non-Invasive Plug-and-Play Inference: At test time, soft memory tokens are concatenated directly alongside system prompts and summaries. The underlying LLM decodes these continuous embeddings without requiring invasive weight modifications.

Benchmarks and Measured Results

REMORY was evaluated across rigorous long-horizon benchmarks including BrowseComp and Terminal-Bench 2.1:

1. 5.2% Footprint Reaches 97.2% Full-Context Ceiling: Consuming just 5.2% of the token sequence length of full-context histories, REMORY retains 97.2% of uncompressed task success rates, dramatically outperforming standard text summaries (68.4%) and embedding retrieval baselines under matching constraints.
2. 41.5% Reduction in Tool Loops and Faults: On Terminal-Bench fault isolation, fine-grained state reconstruction enables agents to accurately recall system file paths and PID configurations, cutting repetitive retry failures by 41.5%.
3. 73.8% Faster TTFT Latency: By shrinking context volume, REMORY slashes Time-To-First-Token (TTFT) by 73.8%, unlocking responsive interactive speeds for enterprise agents.

Getting Started for Developers

REMORY presents an architectural blueprint for production long-horizon agents. Engineering teams building coding agents or persistent monitoring services should transition away from pure text summarization. By combining macro-level narrative summaries with residual vector buffers encoding exact technical parameters and tool outputs, architectures can achieve near-infinite operational horizons while keeping token overheads strictly contained.