Instead of binding nthoughts to costly proprietary speech-scoring platforms, we built our pronunciation evaluation on top of multimodal AI foundation models. Here is how it works.
The problem with traditional pronunciation scoring
Most pronunciation assessment APIs operate on a simple architecture: record audio, upload to a scoring server, receive a phoneme-level breakdown. This creates two problems:
1. Latency: a round-trip to a scoring server and back takes time that breaks conversational flow.
2. Privacy: the audio is transmitted to a third-party server and retained for model training.
Neither of these properties is acceptable for a product whose core promise is privacy-first, real-time feedback.
Voice activity detection (VAD)
Before a spoken utterance can be analysed, it needs to be isolated from ambient noise. nthoughts uses browser-native WebSpeech API for transcription, combined with a VAD layer that detects when the learner is actively speaking versus pausing.
This means the system does not send silence or background noise for analysis, which reduces both computational cost and false-positive correction events.
Multimodal evaluation via Gemini
When a learner completes a speaking turn, a short audio clip (typically 2–8 seconds) is sent over a secure connection to our AI provider's multimodal model endpoint. The model receives both the audio waveform and the target transcript, and it evaluates:
- Vowel timing: are the vowel durations within the expected range for the target pronunciation?
- Word stress: which syllable received primary stress, and does it match the target?
- Speaking rhythm: is the inter-word pacing natural or does it suggest syllable-by-syllable reading?
- Clarity score: an aggregate confidence measure of how well the utterance would be understood by a native listener.
The model returns a structured evaluation — not a raw recording — and the audio clip is not retained after the API response.
What this means for Indian English speakers
Indian English has well-documented phonological features that differ from British or American standard: syllable-timed rather than stress-timed rhythm, different vowel realisations (particularly the TRAP and STRUT distinction), and retroflex consonants.
Our evaluation model is calibrated to distinguish between:
- Pronunciation features that would impede intelligibility in a global professional context (these get flagged for practice)
- Features that are characteristic of Indian English but do not impede intelligibility (these are not flagged — we are not trying to eliminate accent, we are improving intelligibility)
This distinction is important. Accent reduction is not our goal. Clarity for global professional communication is.
Learn more about pronunciation feedback or start a practice session.