Along with it, the company presented a set of Russian-language audio data for testing ASR models under identical conditions
MWS AI, part of MTS Web Services, has released RESON, an open-source tool for evaluating automatic speech recognition models. It allows comparing them across more than 10 metrics and generates an interactive report with analysis results.
To test a model in RESON, its transcripts and reference texts are uploaded. The system unifies the data format, calculates selected metrics, and helps identify typical errors. Available metrics include WER and CER, which evaluate errors at the word and character levels, as well as MWA and the MWS AI-developed WIS metric, which considers the significance of individual words for a specific scenario.
RESON is distributed as an open Python library. It can be run via the command line or API and used within one's own infrastructure, including closed circuits.
Simultaneously, MWS AI introduced the MWS RESON ASR OSD dataset for comparing models under identical conditions. It includes over 12,000 Russian-language audio recordings with a total duration of about 38 hours. The sample contains conversations, phone numbers, dates, amounts, company and service names, as well as recordings with noise and background speech.
Across four datasets, GigaAM v3 from Sber was the best among the tested models with an average WER of 7.42%. Vosk showed 11.07%, Nvidia Parakeet 15.83%, and Nvidia Nemotron 25.52%.
