Artificial Analysis Launches the Diarization Index for Speech to Text: APIs Scored on Word Diarization Error Rate, NVIDIA Nemotron 3.5 Transcribe Leads at 3.3% Ahead of Voxtral Small Transcribe 3 at 7.5%
On November 13 Artificial Analysis published the Diarization Index, a benchmark for whether speech to text APIs attribute each word to the person who actually said it. It scores Word Diarization Error Rate (WDER), the share of words given to the wrong speaker or to no speaker, across 97 English conversations of two to eight people, averaging three speaker bands (2, 3-4 and 5-8) with equal weight. NVIDIA's open-weight Nemotron 3.5 Transcribe leads at 3.3%, ahead of Voxtral Small Transcribe 3 at 7.5%, the most accurate priced API; MAI-Transcribe-2 scores 8.8% and is one of the fastest.
Key Takeaways
- The new metric is WDER = (wrong-speaker words + missed words) / scored reference words, lower is better; AA says it ranks on WDER instead of DER because for APIs DER mostly scores word timestamps rather than who spoke
- The set has 97 conversations of about 2-9 minutes each, about 6.8 hours in total, mixing real and synthetic speech; most audio is proprietary and held out, plus about 126 minutes of public NOTSOFAR-1 meeting recordings
- APIs are called directly with diarization switched on and are not told how many people are talking, with each conversation sent whole; price includes any separate diarization charge
- Nemotron 3.5 Transcribe goes from 1.7% WDER with two speakers to 5.6% with five to eight; in the largest groups every other API gets more than 10% of words wrong
- Voxtral Small Transcribe 3 scores 7.5% at $4.00 per 1,000 minutes; MAI-Transcribe-2 scores 8.8% at $1.67 and transcribes 93.7 seconds of audio per second of waiting; Nemotron 3.5 Transcribe is open-weight and has no listed price
Linked Models & Agents:
⚡ Nemotron 3.5 Transcribe⚡ Voxtral Small Transcribe 3⚡ MAI-Transcribe-2🌐 Artificial Analysis🌐 NVIDIA🌐 Mistral AI🌐 Microsoft
