MSI-Bench tests voice agents with 1,152 multi-party, multi-turn audio scenes split evenly between English and Mandarin. Each case includes participant context, expected tool calls and atomic rubrics for memory, instruction following and reasoning. The strongest setups pass every rubric on only 66.8% of English and 54.5% of Mandarin cases; the best open-weight setups reach 34.0% and 19.3%. Systems also answer when nobody has addressed them.
MSI-Bench finds voice agents lose track of speaker identity and authority
2026-09-22Frontier watchWATCH
MSI-Bench tests voice agents with 1,152 multi-party, multi-turn audio scenes split evenly between English and Mandarin. Each case includes participant context, expected tool calls and atomic rubrics for memory, instruction following and reasoning. The strongest setups pass every rubric on only 66.8% of English and 54.5% of Mandarin cases; the best open-weight setups reach 34.0% and 19.3%. Systems also answer when nobody has addressed them.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For meeting, household and collaborative voice agents, transcription quality alone does not test whether the system knows who spoke, whose instruction has authority or when it should remain silent. Add speaker-scoped tool permissions and negative 'do not respond' cases to acceptance tests. MSI-Bench is public, but it evaluates general multi-speaker interaction rather than any one translation model.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Cases | 1,152 total; 576 English and 576 Mandarin |
|---|---|
| Capabilities | multi-speaker memory, instruction following and reasoning |
| Best Frontier | 66.8% English and 54.5% Mandarin all-rubric pass rate |
| Best Open Weight | 34.0% English and 19.3% Mandarin |
| Failure Split | open weights are limited mainly by speaker perception; frontier systems still miss speaker-scoped decisions and conversational restraint |
| Artifacts | public Hugging Face dataset and GitHub code |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 2
Verification cost 3
Risk 2