MemT-1.0: Towards Memory-Native Evolution Slashing 128K KV Footprint by 66x with Gated DeltaNet-2

Background and the Problem

Modern LLMs and autonomous agents facing extended multi-session tasks rely heavily on external workarounds: either caching full-attention KV matrices (causing quadratic computation and memory consumption running into hundreds of gigabytes at 128K tokens), or using Vector RAG (which frequently suffers from semantic fragmentation and retrieval hallucination). Endowing models with the intrinsic capacity to accumulate, internalize, and continuously leverage experience—analogous to biological memory—represents a fundamental frontier. In arXiv:2610.26825, researchers introduced MemT-1.0, turning memory into an intrinsic compute mechanism.

Architecture and How It Works

1. Gated DeltaNet-2 Recurrent Engine:

• Built upon the Metis framework, incorporating an upgraded Gated DeltaNet-2 architecture with a strictly stabilized within-turn token-level memory update mechanism;

1. Mixture-of-Storage Native Scaling:

• Introduces Mixture-of-Storage, dynamically routing incoming semantic content across specialized recurrent memory slots to break traditional state-capacity saturation;

1. Two-Stage Training Infrastructure:

• Backed by continued pre-training and specialized post-training over a tailored 34B token corpus, releasing the MemT-1.0-Nano and MemT-1.0-Flash model family.

Benchmarks and Measured Results

Evaluated on long-context benchmarks, memory retention, and procedural agentic execution (Figure 1, Section 5):

• 128K Context Horizon Efficiency:
• At a sequence history of 128,000 tokens, MemT-1.0-Nano achieves approximately a 1.86x end-to-end inference speedup over full-attention KV cache baselines;
• Delivers a massive 66x reduction in persistent memory storage, slashing hardware costs for continuous multi-session operation;
• Task Success & Procedural Internalization:
• Outperforms parametric memory baselines while matching or surpassing heavy retrieval-augmented (RAG) pipelines;
• Internalizing procedural experience translates directly into higher task success on complex, multi-stage agentic workflows without context degradation.

Getting Started for Developers

• Research Access: Detailed formulations and evaluations are documented in arXiv:2610.26825;
• Architectural Implications: System designers building continuous coding assistants, personal companions, or site reliability engineering agents should evaluate memory-native architectures as a high-efficiency alternative to unwieldy vector databases and expensive KV cache deployments.