Bilingual ASR Benchmark for Indian Railway PA Announcements
EMNLP 2026 Demo Track
Upload or record a railway announcement and get instant transcription with entity extraction.
Try DemoParticipate in our study evaluating how well ASR transcripts help passengers understand railway announcements.
Participate (~10 min)Task Score = mean(Train No. Acc, Platform Acc, Ann. Type F1)
| System | WER ↓ | TN ↑ | Plat ↑ | AnnF1 ↑ | Task ↑ | Lat(s) ↓ |
|---|---|---|---|---|---|---|
| — Commercial API — | ||||||
| Google STT Chirp2 | 45.0 | 78.5 | 83.0 | 76.3 | 79.3 | NA |
| — Open-source baselines — | ||||||
| whisper-large-v3 | 35.0 | 63.3 | 70.6 | 75.3 | 69.7 | 56.2 |
| qwen3-asr-1.7B | 28.9 | 79.0 | 83.6 | 78.7 | 80.4 | 45.4 |
| whisper-small | 52.1 | 40.3 | 64.4 | 65.1 | 56.6 | 11.5 |
| conformer-large | 58.1 | 48.6 | 56.0 | 64.7 | 56.4 | 3.4 |
| qwen3-asr-0.6B | 35.9 | 69.9 | 84.6 | 75.5 | 76.7 | 21.3 |
| — LoRA fine-tuned + DB-aware PP — | ||||||
| whisper-small | 39.2 | 66.4 | 74.0 | 69.1 | 69.8 | 11.8 |
| conformer-large | 55.7 | 49.6 | 49.8 | 62.9 | 54.1 | 3.7 |
| qwen3-asr-0.6B ★ | 33.2 | 76.3 | 82.4 | 80.5 | 79.7 | 21.6 |
★ Best system · Whisper variants use faster-whisper (CPU). Lat(s) = avg inference time on CPU.