Open ASR Leaderboard Adds Egyptian and Nigerian English: What Changes for Speech-to-Text Tests

The Open ASR Leaderboard now tests two more English accents with data models cannot have trained on. On November 6, 2026, Hugging Face announced a collaboration with Liva AI that adds Egyptian English (1.34 hours) and Nigerian English (1.55 hours) as private, held-out conversational test sets. The Private tab also gets a new accent breakdown: US versus non-US, and Inner Circle versus Outer and Expanding Circles. The leaderboard's headline average does not change.

If you pick a speech-to-text model for a product with global users, this is a useful change to read. It is small in hours of audio, but it tests the thing that public benchmarks tend to hide: how a model handles accents it was not tuned for. The announcement is on Hugging Face.

TL;DR: what changed and what did not

What is the Open ASR Leaderboard?

It is Hugging Face's public ranking of speech recognition models by word error rate (WER) and speed. According to Hugging Face's May 2026 post, it launched in September 2023 and has been visited over 710,000 times. Two ideas hold it together, in the team's words: standardization (all test sets gathered on the Hub, and a normalizer based on Whisper's that removes punctuation and casing and maps to American spelling) and openness (UI code and evaluation scripts are open source).

Both help and hurt. Standard, open tests make results comparable. They also make it easy to tune a model to the test, a habit the team calls "benchmaxxing." A model can score well and still fail on real audio. That tension is why private data keeps coming up.

Why does private test data matter?

In May 2026, the team published "Adding Benchmaxxer Repellant to the Open ASR Leaderboard", opening with Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." Appen and DataoceanAI supplied 11 private splits covering scripted and conversational speech in five accents, and Hugging Face kept them hidden to reduce the risk of test-set contamination.

Two design choices from that post matter for how you read the new Egyptian and Nigerian sets:

• The default average stays public-only. The post said Hugging Face was not updating the average WER; users could toggle private datasets in to see the effect.
• Vendors that sell training data are a risk, one commenter argued. In the comments, a reader (weiwchu) raised the concern that the two data vendors also offer datasets to ASR providers for training. Hugging Face's Eric Bezzam replied that the private data is excluded from the default average WER.

Liva AI is a third provider. Hugging Face describes it as a company that collects multimodal interaction data through consumer products used by real people, with natural, high-context behavior rather than artificially elicited tasks. That is the company's own description. Adding a new provider also fits the team's stated mitigation: use several providers so no one source gives a lasting edge.

What is new in the November update?

Open ASR Leaderboard private data accent breakdown by English accent Image: Hugging Face

The two new sets are both conversational, with transcripts that keep punctuation, casing and disfluencies (the "um" and false starts of real speech). That makes them harder and closer to call audio than a scripted read-aloud.

The accent breakdown is the other half of the update. Using Kachru's Three Circles of English, the leaderboard will now report:

1. US versus non-US, where non-US means UK, Australia, Canada, India, Nigeria and Egypt.
2. Inner Circle versus Outer and Expanding Circles. The Inner Circle is the US, UK, Australia and Canada. The Outer and Expanding Circles are India, Nigeria and Egypt.

Hugging Face credits Hume AI's leaderboard for the idea, and says it will split the Outer and Expanding Circles further as more accents arrive. Hugging Face says Indian English joined earlier through a private split made with Voice Arena.

How should I read the numbers when they appear?

Treat the new sets as a stress test, not a verdict.

• Small samples. 1.34 and 1.55 hours of audio is a few speakers at most. A model that looks great or poor here may owe it to one voice, one microphone or one topic. The post does not say how many speakers each set has.
• Conversational only. There is no scripted counterpart yet for these accents, so you cannot separate "accent" from "spontaneous speech." A drop could come from either.
• One metric. WER hides a lot. Names, numbers and code-switching can matter more to your product than the average.
• No scores in the post. The announcement does not name top models or publish scores. Go to the leaderboard Space for results, and remember the post gives no information on who leads.

A practical test: pick the three models you are considering, run them on 20 minutes of your own audio from your actual users, and compare. Use the leaderboard to shortlist, not to decide.

Why does this matter for people who build with voice?

Voice products are shipping fast. We have covered new speech-to-text launches from OpenAI's GPT Live Transcribe, Google's Gemini 3.5 Transcribe, Grok Voice Transcribe 2 and Meta Muse Voice transcribe. Each comes with vendor-chosen numbers. A third-party leaderboard with held-out data in under-represented accents is one of the few checks that the vendor does not control.

Two cautions for builders:

• Accent coverage is still narrow. Two more accents is progress, not coverage. Hugging Face itself calls it "continued effort." If your users speak Hindi-English, Arabic dialects or Nigerian Pidgin, test directly.
• Private does not mean unbeatable. If a model vendor buys data from the same providers, the edge can come back over time. This is the concern a commenter raised in May; the default average excludes private data.

For a wider view of why leaderboards drift and what to do about it, see our explainers on Goodhart's Law and benchmark contamination and Google DeepMind's double-blind evaluation work. If you work with on-device dictation, our look at Superwhisper's S1 Mini transcript normalizer shows how much the text-normalization step alone can change WER-style results.

What is word error rate, and why do accents move it?

Word error rate counts the words a model gets wrong (substitutions, deletions and insertions) divided by the number of words in the reference transcript. A WER of 10 percent means roughly one word in ten is off. Because the leaderboard normalizes both the model output and the reference (no punctuation, no casing, American spelling), it compares words, not formatting.

Accents move WER for simple reasons. A model learns the sounds, rhythms and vocabulary it saw in training. If most of its training audio is American English, other accents land further from what it knows, and errors rise. Spontaneous speech adds fillers, restarts and overlapping talk, which scripted audio lacks. The new sets combine both pressures: non-US accents and unscripted conversation.

The May post makes a point worth repeating here. In the team's words from their report, there is no single catch-all ASR model: some do better on American English, others on diverse accents and multilingual settings, others are tuned for speed or conversational audio. A model that scores lower on one axis is not necessarily a worse model overall. So the right question is not "which model is first" but "which model is best on the audio my users produce."

How can I contribute or follow along?

• Use the leaderboard: open the Open ASR Leaderboard Space on Hugging Face and switch to the Private tab to see the accent breakdown.
• Suggest a dataset or model: the open_asr_leaderboard GitHub repository is the place to propose additions; check its contribution notes first.
• Read the method: the May post explains the normalizer, the scripted versus conversational splits and why the average is public-only.

What to do this week

1. If you run ASR in production with African or Middle Eastern English speakers, add the Private tab to your model-selection checklist.
2. Build a 20-minute internal test set from consented, real user audio, and score candidates on it.
3. Log WER by accent group, not only overall. A good average can hide a bad tail.
4. Re-check in a few weeks. Hugging Face says more accents will join and the Outer and Expanding Circle group will be split.

Related reading

• OpenAI GPT Live Transcribe and GPT Transcribe
• Gemini 3.5 Transcribe launch
• Grok Voice Transcribe 2
• Meta Muse Voice transcribe
• Superwhisper S1 Mini and transcript normalization
• Goodhart's Law and AI benchmark contamination
• Official: Egyptian and Nigerian English join the Open ASR Leaderboard · Adding Benchmaxxer Repellant

Accurate as of November 6, 2026. The announcement gives no model scores; check the leaderboard Space for current results.