Omni-Embed-Mini Binds 6 Modalities Without Forgetting: MBZUAI Opens 0.9B Dense Distillation Model Outperforming Gemini

核心背景与行业痛点

Multimodal RAG and agent memory repositories require indexing cross-modal evidence (PDF layouts, audio recordings, video frames, text notes). Extending text embedding models to diverse modalities typically triggers catastrophic forgetting in dense text retrieval. Meanwhile, existing omni-modal embedding models deploy heavy multi-billion parameter backbones that inflate latency and hardware budgets in production embedding pipelines.

架构亮点与底层机制

MBZUAI researchers introduce Omni-Embed-Mini, a compact 0.9B architecture bridging 6 distinct modalities:

1. Self-Distillation via Cascaded Captions: Pairs media assets with rich descriptive captions and targets the frozen text backbone's own embedding representation, establishing bit-identical geometric alignment without auxiliary models.
2. Bit-Identical Text Weights: Keeps 100% of text parameters frozen throughout training, guaranteeing zero performance regression on text benchmarks (49.57 nDCG@10 on MTEB-v2 BEIR-8).
3. Matryoshka SigLIP Formulation: Integrates adaptive Matryoshka dimension truncation with SigLIP objectives, facilitating flexible downstream embedding compression.
4. Online Dynamic Hard Negative Mining: Progressively refines negative pairs as modality projectors evolve, enhancing cross-modal discrimination.

权威 Benchmark 与实测跑分对比

Benchmarked across six modalities covering text, vision, audio, speech, and video retrieval:

1. 2.7x to 9.5x Smaller Footprint: The 0.9B model preserves flawless text retrieval while unifying five extra modalities, drastically undercutting open competitors in parameter scale.
2. Surpasses Gemini-Embedding-2: The 2.3B model variant outperforms Google's proprietary gemini-embedding-2 across the cross-modality aggregate benchmark.
3. >18% Retrieval Gain on Rich Documents: Cascaded supervision yields an 18.6% recall jump over standard SigLIP baselines on visually-rich PDFs and long video slices.

开发者实战落地与开箱指南

Omni-Embed-Mini models and evaluation harnesses are available on Hugging Face and GitHub. Compatible with standard Sentence-Transformers APIs, practitioners can integrate single-model omni-modal vector search directly into local and cloud RAG pipelines.