When I re-evaluated the speech recognizer for my iPhone diary app, Apple's recognizer looked best on the commonly used read-speech benchmark. When I re-measured on conversational audio, which is what the app actually handles, Apple's character error rate was about 1.8× that of Parakeet, the model I was already using. Had I switched based on the benchmark, transcription quality would have dropped. Here is how I compared them, where I almost went wrong, and what I chose, with the numbers.
What I wanted to do
I build Liner Notes, an iPhone diary app that turns conversations into text on the device. For transcription I was using NVIDIA's Parakeet Japanese model through FluidAudio on Core ML.
In October 2026, Apple's recognizer (iOS SpeechTranscriber), Mistral's Voxtral and IBM's Granite Speech all became usable for Japanese, so I compared them to see whether switching was worth it. The requirements: it runs on the iPhone, it is fast enough to process in the background after recording, and it transcribes conversation accurately.
How Japanese speech recognition is evaluated
Japanese has no spaces between words, so instead of word error rate (WER) the usual measure is character error rate (CER). You align the reference and the output character by character and divide substitutions, deletions and insertions by the number of reference characters. I normalized both with NFKC and removed punctuation, spaces and symbols before comparing. Notation differences such as 三十分 vs 30分 still count as errors.
Public Japanese evaluation sets fall roughly into read speech and spontaneous or conversational speech.
| Dataset | Type | Contents |
|---|---|---|
| FLEURS (Google) | Read | The same sentences read in many languages. Often used for published model results |
| Common Voice (Mozilla) | Read | Sentences read by volunteers |
| JSUT | Read | Sentences read by a single speaker |
| CSJ (Corpus of Spontaneous Japanese) | Spontaneous | Lectures and similar talks. Paid |
| ReazonSpeech | Conversation | TV broadcast audio (drama, variety, news) with captions |
Read speech first, and I was ready to switch to Apple
I started with FLEURS Japanese (dev set, 266 clips, 56.5 minutes). Voxtral's published results use the same data, so it seemed easy to compare.
All models ran on the same Mac (Apple silicon, 24 GB).
- Parakeet: FluidAudio 0.17.4. As in the app, segments are cut at quiet points every 6–12 seconds and padded with 0.5 s of silence on each side
- Apple: SpeechTranscriber (ja-JP)
- Voxtral: 6-bit quantized Voxtral Mini 4B Realtime on the Mac GPU, at two latencies: 480 ms (default) and 2400 ms (most accurate)
| Model | CER | Time for 56.5 min |
|---|---|---|
| Voxtral (2400 ms) | 3.61% | 2,067 s |
| Apple | 4.68% | About 69 s |
| Parakeet (in use) | 5.29% | 18.8 s |
| Voxtral (480 ms) | 7.02% | 3,003 s |
CER on read speech (FLEURS Japanese)
% (lower is more accurate)
Show as a table
| Item | Value |
|---|---|
| Voxtral (2400 ms) | 3.61% |
| Apple | 4.68% |
| Parakeet | 5.29% |
| Voxtral (480 ms) | 7.02% |
Voxtral at 2400 ms was the most accurate, but about 110× slower than Parakeet and 3.6 GB even at 6 bits, so running it in the background on an iPhone is not realistic. Apple made fewer errors than Parakeet, and since it is built into the OS, the app would not need to ship a model. At this point I was planning to switch to Apple.
Measured on conversation, Apple's error rate was about 1.8×
FLEURS, though, is clear reading in a quiet room. What the app actually handles is conversation, with hesitations, backchannels and overlapping speech. So I measured again under the same conditions on the ReazonSpeech test set, which is TV broadcast conversation (2,632 clips, 258 minutes).
| Model | CER | Clips with no output | Time for 258 min |
|---|---|---|---|
| Parakeet | 12.74% | 0 | 118 s |
| Apple | 22.96% | 37 | 631 s |
Apple and Parakeet swap places between read speech and conversation
CER (%, lower is more accurate)
Show as a table
| Parakeet | Apple | |
|---|---|---|
| Read (FLEURS) | 5.29% | 4.68% |
| Conversation (ReazonSpeech) | 12.74% | 22.96% |
The ranking flipped. Looking at Apple's errors, it returned nothing for 37 short clips and sometimes added polite phrases nobody said, such as "thank you very much" and "nice to meet you." It is the same kind of error that a beta tester had reported: a "hello" that nobody said showing up in the transcript. A greeting nobody said, left in someone's diary, is a real problem.
Apple's time includes starting up for each clip, so the speed gap is smaller than the table suggests. But for conversation accuracy, Parakeet was the one to choose.
Adding Granite: 0.5 s of silence moved the number by 3 points
Next I added IBM's Granite Speech 5.0 TurboCTC (470M parameters), which gained Japanese support on October 21. The first result on conversation was Granite 9.79% vs Parakeet 12.74%. Now it looked like I should switch to Granite.
But Parakeet's clips were padded with 0.5 s of silence on each side, as in the app, and Granite's were not. Here are the results with matched conditions (Granite converted to Core ML for the iPhone).
| Condition | Granite | Parakeet |
|---|---|---|
| Conversation (ReazonSpeech), no padding | 9.99% | 11.73% |
| Conversation (ReazonSpeech), 0.5 s padding | 13.03% | 12.74% |
| Continuous audio (about 4 min × 14), 0.5 s padding | 4.36% | 4.39% |
Padding changes the gap between Granite and Parakeet
CER (%)
Show as a table
| Granite | Parakeet | |
|---|---|---|
| Conversation, no padding | 9.99% | 11.73% |
| Conversation, 0.5 s padding | 13.03% | 12.74% |
| Continuous audio, 0.5 s padding | 4.36% | 4.39% |
ReazonSpeech clips contain fragments of the neighboring utterances that are not in the reference captions. Granite does not transcribe fragments at the very edge of its input; with padding they are no longer at the edge, so it transcribes them, and that looks like more errors. On segments cut from long continuous recordings, which is what the app actually handles, there is no difference between the two.
Finally, reading my own recordings side by side
With no difference on public data, I read the two transcripts side by side for 6 of my own recordings (2 h 10 min in total).
| What I looked at | Granite | Parakeet |
|---|---|---|
| Fast, overlapping speech | Drops phrases | Rarely drops them |
| Backchannel-only segments (11) | Wrote none of them | Wrote them |
| Japanese written as an English sentence | 3 segments | None |
| Numbers and Latin-script names | Accurate | Sometimes garbled |
Backchannels and overlapping speech matter in a record of a conversation. Granite was better on numbers and names, but overall Parakeet missed less.
Conclusion: stay with Parakeet
| Model | Good at | Weak at | Speed |
|---|---|---|---|
| Parakeet | Conversation, backchannels, overlapping speech | Slightly behind Apple on read speech | Fast (258 min in 118 s) |
| Apple | Read speech; no model to ship | Returns nothing on conversation, adds greetings nobody said | Medium (258 min in 631 s) |
| Granite | Speed, numbers and Latin-script names | Backchannels, overlapping speech, switching to English | Fastest (10 min in 2.1 s on iPhone 16) |
| Voxtral | Read-speech accuracy (2400 ms) | Slow, large model (3.6 GB) | Slow (56.5 min in 2,067 s) |
The app stays on Parakeet. It is the most accurate on conversation and does not drop backchannels. I kept the Core ML conversion of Granite as a fallback in case speed becomes a problem.
Summary
- Compare speech recognizers on evaluation data close to the audio your app handles. For conversation, use conversational audio such as ReazonSpeech. Read speech (FLEURS) ranked them the other way round
- Match the preprocessing: how segments are cut and how much silence is padded. 0.5 s of silence moved the error rate by 3 points
- If public data shows no difference, read transcripts of your own recordings side by side. Missing backchannels and greetings nobody said barely show up in the error rate
More engineering notes
- On iOS, run background audio processing on the Neural Engine
In an iOS BGContinuedProcessingTask, CPU work ran 12-21x slower while Core ML on the Neural Engine stayed within 2x. Three causes and the fix, measured.
- Same transcription accuracy, three times the errors in the summary
Two recognizers with the same CER produced 3x the summary errors. Why a stricter summary prompt made it worse, and how to compare by summary quality.
- Apple's Private Cloud Compute refused 10.6% of my app's requests as unsafe
Apple's Private Cloud Compute (Foundation Models) refused 10.6% of 614 production requests and rate-limited 9.6%. What broke and how I redesigned.
- I built an AI copy of myself following prior research. Whatever I fed it, agreement stopped at 62–67%
Following prior research, I built an AI copy of myself. Diary or one line, agreement stayed at 62-67%. What identifies a person is how they speak.