Adobe & Brown University Release OmniXRef: Unified Multi-Reference Video Editing Slashing Camera Trajectory Error by 5x

Background and the Problem

Video editing is the core engine of digital filmmaking, advertising production, and visual storytelling. However, conveying complex cinematic intent through pure text instructions is fundamentally difficult. A creator often wants to "preserve the actor's actions and background from the original footage, but replicate the drone camera arc from Reference Video A, transfer clothing patterns from Reference Image B, and apply lighting from Reference Image C." Conventional models handle only isolated single-turn edits, suffering severe attribute entanglement when presented with multimodal references. Adobe Research and Brown University introduced OmniXRef (arXiv:2610.22744) to solve unified multi-reference editing.

Architecture and How It Works

1. Unified Reference-Bank & Single-Stream Diffusion Transformer:

• Introduces a flexible Reference Bank interface where arbitrary combinations of text instructions, static images, and reference videos are interleaved with source video latents into a unified sequence;
• A single-stream Diffusion Transformer processes all reference contexts in shared self-attention, preserving source composition while transferring target attributes;

1. Mix-Patchify & Reference-Relative RoPE (R2-RoPE):

• Utilizes Mix-Patchify to dynamically balance token distributions across heterogeneous image resolutions and aspect ratios;
• Deploys R2-RoPE (Reference-Relative Rotary Positional Embeddings), allowing seamless generalization to variable reference counts without positional ambiguity;

1. Curriculum Learning Across Five Attribute Axes:

• Progressively scales task complexity and video resolution across Camera, Motion, Content, Style, and Visual Effects (VFX).

Benchmarks and Measured Results

Evaluated on the newly introduced OmniXRef-Bench against specialized open-source tools and commercial models (Tables 4–5, Section 5.2):

• Camera Trajectory Transfer Accuracy (Table 5):
• OmniXRef achieves a Rotation Error (RotErr) of 0.874, Translation Error (TransErr) of 3.961, and CamMC of 4.437;
• Decisively outperforms specialized SOTA CamCloneMaster (2.086 / 9.329 / 10.332);
• Slashes trajectory error by more than 5x compared to leading commercial engines such as Seedance 2.5 (RotErr 4.373 / TransErr 22.445 / CamMC 24.001);
• Multi-Attribute Aesthetic Scores (Table 4):
• Scores 4.49 in Content (vs MiniMax-H3's 2.23 and RefVFX's 1.98), 3.75 in Style, and 3.55 in VFX, establishing the highest composite score across open-source generative architectures.

Getting Started for Developers

• Project Page: Accessible at oliver-cong02.github.io/OmniXRef.github.io/, with complete details in arXiv:2610.22744;
• Production Pipeline Implications: OmniXRef provides a clean blueprint for integrating AI into professional nonlinear video editors. By enabling granular reference assignment (e.g. video for camera motion, image for costume), creators can achieve deterministic creative direction previously impossible with prompt-only video generators.