Are Embeddings Overpriced? Frozen 16 MB Vocabulary Slashes Small Model Weights by 62% with Zero Downstream Accuracy Loss

Background and the Problem

As Small Language Models (SLMs, e.g., Gemma 3 270M) and ternary neural networks (BitNet b1.58) expand onto edge microcontrollers and wearable devices, binary footprint and runtime VRAM become primary constraints. However, existing SLMs suffer from an absurd structural paradox: the embedding table dominates the vast majority of the model's parameters. In Google's Gemma 3 270M, the vocabulary table accounts for 167.8M out of 268.1M parameters (62.6% of the entire model). Because embeddings are sensitive to quantization, they remain in 16-bit floats while transformer layers are quantized to 1 or 2 bits, negating edge compression gains. A pioneering study (arXiv:2610.22428) dismantles this bottleneck.

Architecture and How It Works

1. Frozen 16 MB Binary Codebook:

• Discards trainable float embedding tables entirely. Pre-computed embeddings from a reference model undergo principal component projection and sign binarization, yielding a single 16 MB frozen table of 512-bit binary codes with zero trainable parameters;

1. Lightweight Linear Projection Head:

• Tokens read directly from the static 16 MB binary lookup table and pass through a compact linear projection into the hidden dimension, decoupling vocabulary capacity from parameter storage.

Benchmarks and Measured Results

Empirical pretraining evaluations on Gemma 3 270M (Tables 1 & 2):

• 62.3% Collapse in Trainable Parameters (Table 1):
• Standard float model: 268.1M parameters (167.8M / 62.6% tied in embeddings, 70.8M FFN, 29.5M attention);
• Frozen codebook model: Trainable parameter count collapses to 101.0M parameters (-62.3% reduction);
• Zero Downstream Degradation (Table 2):
• HellaSwag: Matches the baseline exactly at 30.4% vs 30.4%;
• LAMBADA: Rises from 21.6% to 23.2% (+1.6 percentage points);
• PIQA: Increases from 60.6% to 61.5% (+0.9 percentage points);
• ARC-Easy: Scores 42.5% (vs 45.9%);
• Ternary Model Synergy:
• In a 1.58-bit ternary configuration with frozen embeddings (101.2M parameters), the model achieves 22.5% LAMBADA, 29.4% HellaSwag, and 59.5% PIQA, demonstrating natural synergy with extreme quantization.

Getting Started for Developers

• Reference: Detailed derivations are provided in arXiv:2610.22428.
• Edge Engineering Takeaways: When deploying SLMs to embedded systems or browser WebAssembly instances, replacing trainable embedding tables with a 16 MB PCA-binarized lookup table eliminates over 60% of storage and memory demands without compromising downstream language comprehension.