Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,411 words · 8 segments analyzed
Speech recognition models have become remarkably good at transcribing clean audio recorded under controlled conditions. But real dictation rarely happens under those conditions. Millions of people use Wispr Flow to message friends, write emails, code, and work through ideas at their desks, between meetings, during commutes, and in busy offices. They speak through laptop microphones, earbuds, and headsets, often with other voices, music, or traffic in the background.Today, the Wispr Advanced Interfaces Lab is introducing Canto, our latest speech model for real-time dictation. On an evaluation of real-world dictations, Canto achieved the lowest word error rate among all the models we tested. We compared Canto with models from Google, OpenAI, AssemblyAI, and Deepgram.Canto is the first model in a broader research and development program at Wispr Advanced Interfaces Lab. In this post, we share how it performs, how we trained it to handle challenging real-world conditions, and the research already shaping what comes next. “Today, the Wispr Advanced Interfaces Lab is introducing Canto, our latest speech model for real-time dictation.” Ariya Rastrow CSO, Wispr Flow Evaluating Canto in real-world conditionsTo test Canto in real-world conditions, we created an evaluation set composed of 10 hours of English-language Wispr Flow dictations from more than 2,300 unique speakers, randomly sampled across applications and use cases. We were careful to enforce a strict separation between speakers represented in the train and test sets to avoid overfitting on speaker characteristics. Every sample came from a user who opted in to Wispr’s data-sharing setting, which allows their data to be used anonymously to evaluate and improve our models. More information about this setting is available in our data-controls documentation.Canto achieved the lowest Word Error Rate (WER) of the models in our comparison. WER measures word substitutions, omissions, and insertions relative to a human-transcribed reference (lower is better).Word Error Rates on 10 hours of Wispr Flow dictations, randomly sampled across applications and use cases.Performance under the most challenging conditionsThe model performed well on randomly sampled data, but we were also interested in studying its performance in the most challenging situations. We built a separate, 3-hour challenge evaluation set that featured those conditions most likely to cause dictation to fail.The set includes audio affected by nearby speech, music, traffic, wind, low recording volume, and whispered or far-field speech. It also includes short dictations that give a model very little surrounding context and language to help resolve ambiguity. On the full challenge set, Canto ranked second behind Gemini 3.1 Pro, a much larger frontier-size multi-modal model that is not suitable for real-time low-latency applications. Among the real-time transcription models we evaluated, Canto achieved the lowest WER.Word Error Rates on a 3-hour audio challenge dataset, assembled to stress-test transcription models under the most difficult real-world conditions.We further examined this dataset to understand how the models behave differently. Gemini 3.1 Pro achieved the lowest WER on noisy audio. Canto tied for the lowest WER on low-volume speech and short dictations. Short dictations produced the highest error rates across the comparison. A single mistake has a larger effect on WER when an utterance contains only a few words, and the models also have less linguistic context available to resolve ambiguity.Word Error Rates across three subsets of the audio challenge set. Noisy audio was defined by low SNR dictations where there was significant background noise (i.e. traffic, wind, competing background speech). Low volume samples were made up of whispered or far-field speech. Short dictations were 1-2 word samples where there was little surrounding context.Comparing performance on public benchmarksWe also evaluated Canto on three public English datasets: LibriSpeech, FLEURS, and Common Voice. Canto tied for the lowest WER on LibriSpeech. It remained competitive on FLEURS and Common Voice, although it did not lead either evaluation.Word Error Rates on the English subsets used in our evaluation: FLEURS, LibriSpeech, and Common Voice. Models were evaluated without contextual prompting.
These public datasets provide useful, reproducible comparisons, but they largely contain read speech. LibriSpeech is drawn from audiobooks, while FLEURS and Common Voice consist primarily of people reading prepared sentences. They capture a different distribution from everyday “in-the-wild” dictation, where people speak spontaneously, pause, revise their thoughts, and often provide very little linguistic context.
The contrast helps explain why we evaluate Canto on both public datasets and real Wispr usage.Post-training Canto with Supervised Fine-Tuning and Reinforcement LearningCanto starts from a model pretrained on millions of hours of speech and text. We then train Canto in two stages. First, we show it audio paired with reference transcripts.
The model learns to predict the words in each transcript, one step at a time. This stage, called supervised fine-tuning, teaches it how to perform the transcription task. Next, we train it to compare the quality of complete transcripts. For the same audio, the model generates several possible transcriptions. We score each one against a reference, then use those scores to make better transcriptions more likely in future training. This is known as reinforcement learning, or RL.The distinction is in how the model receives feedback. Supervised fine-tuning provides the expected words at each step. Reinforcement learning evaluates the completed transcription (generated by the model). That lets us train around the outcomes we care about, such as reducing recognition errors under difficult recording conditions. Our approach uses Group Relative Policy Optimization, or GRPO. The central idea is simple: compare the candidate transcripts within each group and learn from their relative scores. For each audio example, we sample several candidate transcripts from the model being trained. We call these candidates rollouts. Each receives a reward: a numerical score measuring how well it satisfies the training objective.GRPO compares each candidate’s reward with the rewards of the other candidates for the same audio. The resulting relative score, called an advantage, indicates whether that candidate performed better or worse than its group. These advantages guide the training update.Sequence-level training has a long history in speech recognition. Researchers have previously optimized ASR systems using minimum word error rate training and policy-gradient methods.
GRPO was introduced more recently in DeepSeekMath, and direct applications to autoregressive speech recognition have only begun to appear in recent work on GRPO for ASR and speech-domain adaptation.“We built infrastructure that can generate and score speech rollouts at scale, allowing us to construct training environments around specific behaviors and failure modes.”For Wispr, the value of RL is not just a lower aggregate error rate. We built infrastructure that can generate and score speech rollouts at scale, allowing us to construct training environments around specific behaviors and failure modes. That system now supports experiments in contextual speech recognition, personalization, diarization, and difficult audio conditions. The research described below extends beyond the training used for the released Canto model and is already informing the next generation of our models.Learning from correctionsThe usage of names and vocabulary changes faster than speech models can be retrained.
A word such as “Claude” might once have been an uncommon alternative to “cloud.” As its usage changes, a practical dictation model needs a way to learn the distinction.For users who have opted into data sharing, corrections can provide a valuable signal for improving recognition. But not every edit points to a transcription error. People also change formatting, rewrite sentences, or simply change their minds.A hypothetical example below illustrates this problem.
The model transcribed “Claude” as “cloud,” which the person corrected. They also changed their mind and rewrote the end of the sentence. Only the first change represents a recognition error.A hypothetical example. The local correction from “cloud” to “Claude” is retained, while the unrelated rewrite is discarded. We can selectively graft the ASR error into the training target for RL.We use signals from the audio and the edit itself to identify corrections that are most likely to represent recognition errors.
These include forced-alignment confidence, which measures how well candidate words match the audio, and the location and shape of the edit. We then graft only the likely correction into the original transcript, leaving the rest unchanged.The resulting transcript becomes the reference used to score GRPO rollouts. A hypothesis receives a higher reward when it makes the intended correction. Because GRPO centers the advantages within each group, words shared by higher-reward and lower-reward hypotheses receive offsetting signals. This does not provide perfect token-level credit assignment, but it makes the strongest contrast more local to the decision we care about. Grafting turns a noisy document-level edit into a more focused sequence-level training signal while preserving the on-policy nature of GRPO.Illustrative group of six model-generated transcripts.