Two speech recognizers had the same character error rate (CER), so I assumed either would give the same quality. When I generated summaries from each transcript and compared them, the number of topics containing an error differed by a factor of three. When I tightened the summary prompt to reduce errors, errors went up. Here is what I should have been measuring and how I compared them again.
What I wanted to do
Liner Notes, my iPhone diary app, transcribes conversations on the device and saves them as summaries, one per topic. People read the summary; they rarely read the full transcript.
When I compared speech recognizers, Parakeet and Granite had almost the same CER under matched conditions (4.39% and 4.36%). I assumed the summaries would be equally good with either.
What CER counts and what it does not
CER aligns the reference and the output character by character and divides substitutions, deletions and insertions by the number of reference characters. Every character weighs the same, so dropping one "uh-huh" backchannel and getting one character of a client's name wrong are both a single-character error.
Only the second one affects the summary. A missing backchannel does not change a summary, but a wrong name or number goes straight into it.
Comparing the summaries
I transcribed the same two meeting recordings with both recognizers and generated per-topic summaries from each transcript. With the recognizer hidden, I sorted every topic into "correct," "contains an error" or "unnecessary" (small talk unrelated to the meeting's purpose).
| Recognizer | CER (public continuous audio) | Topics | Topics with an error | Unnecessary topics |
|---|---|---|---|---|
| Granite | 4.36% | 32 | 9 | 5 |
| Parakeet | 4.39% | 29 | 3 | 4 |
Share of summary topics with an error, and unnecessary ones
% of topics
Show as a table
| Granite | Parakeet | |
|---|---|---|
| Topics with an error | 9/32 (28%) | 3/29 (10%) |
| Unnecessary topics | 5/32 (16%) | 4/29 (14%) |
The CERs were almost identical, yet topics with an error differed threefold. Granite often drops phrases where fast speech overlaps; I believe that broke the flow of the conversation and led to mix-ups in the summary.
Trying to reduce errors, and adding more
To reduce errors, I made the summary prompt stricter. For each topic the model had to cite the transcript lines it was based on, say whether it was the main subject or small talk and how certain it was, and "use only numbers and names that appear in the transcript." Small talk and low-certainty topics were dropped.
| Summary prompt | Topics | Topics with an error |
|---|---|---|
| Original (summarize reasonably) | 30 | 5 |
| Stricter | 22 | 9 |
The stricter prompt raised the share of topics with an error
%
Show as a table
| Item | Value |
|---|---|
| Original (summarize reasonably) | 5/30 (17%) |
| Stricter | 9/22 (41%) |
Errors went up. Looking inside, these were the new errors:
- Copying misheard numbers verbatim: "two or three companies" became "23 companies"
- Packing numbers from another topic into one topic and confusing "100,000" with "1,700,000"
- When told to drop small talk, also dropping the main subject (a discussion of firmware power consumption)
| What I tried | Goal | Result |
|---|---|---|
| Use only numbers in the transcript | Fewer number mix-ups | Copied misheard numbers; more errors |
| Cite the supporting lines | Fewer unsupported topics | Packed numbers into one topic and mixed them up |
| Drop small talk | Fewer unnecessary topics | Less small talk, but the main subject went too |
| Drop low-certainty topics | Fewer topics with an error | Topics fell from 30 to 22 and the error share rose |
Tell the model to "use only numbers in the transcript" and it trusts the transcript's mistakes too. The original, looser prompt read the numbers correctly from context.
The recognizer's per-word confidence did not help either. Parakeet's confidence sits around 0.95 whether a word is right or wrong, so it gives no clue to the errors.
What I ended up doing
I changed the yardstick to "does the summary contain errors or unnecessary information." Whenever I change the recognizer or the summary prompt, I generate summaries from the same conversations and compare them blind before deciding. Around 30 topics is enough to see a difference.
I also changed how summaries are stored. They used to be rewritten into prose; now they are saved as a list, one item per topic:
- Topic (who spoke) [decided] / content
Prose is written only when the diary draft is made. With the numbers kept per topic instead of melted into prose, mix-ups are easier to spot when reading back. People can long-press an item to fix or delete it.
Three things to try next: generate the summary two or three times and keep only the topics that agree, have the model round numbers ("about 20 companies"), and keep low-confidence items with a "check this" mark.
Summary
- Compare speech recognizers on what people actually read. If they read summaries, compare summary errors, not CER. With the same CER, summary errors differed threefold
- Do not tell the summarizer to "use only numbers in the transcript." It copies the mishearings. Asking it to round is safer
- Check the distribution of per-word confidence on your own data before using it. With Parakeet it was no clue to errors
- Compare blind, without knowing which result came from which system
More engineering notes
- On iOS, run background audio processing on the Neural Engine
In an iOS BGContinuedProcessingTask, CPU work ran 12-21x slower while Core ML on the Neural Engine stayed within 2x. Three causes and the fix, measured.
- Choosing a speech recognizer by benchmark alone is risky. Measure with the audio your app actually handles
On read speech Apple's recognizer looked best; on conversation its error rate was 1.8x Parakeet's. How to compare Japanese speech recognizers, with data.
- Apple's Private Cloud Compute refused 10.6% of my app's requests as unsafe
Apple's Private Cloud Compute (Foundation Models) refused 10.6% of 614 production requests and rate-limited 9.6%. What broke and how I redesigned.
- I built an AI copy of myself following prior research. Whatever I fed it, agreement stopped at 62–67%
Following prior research, I built an AI copy of myself. Diary or one line, agreement stayed at 62-67%. What identifies a person is how they speak.