Building a Voice AI for Mutual Funds, and the Browser Speech API Wall I Hit, Abhishek Mukherjee

I set out to build a small thing over a couple of evenings: a voice assistant that sits on a mutual-fund website and answers visitor questions out loud, grounded in the company's own documents. Gemini for the brain, RAG for the data, and the browser for voice so it would cost almost nothing to run. The RAG part came together quickly. The voice part is where I ran into a wall that took a while to even understand, let alone fix.

The Idea

The plan was deliberately boring and cheap. No fine-tuning, because for factual Q&A over a set of documents, retrieval beats fine-tuning almost every time and you can update the data without retraining. So the shape was:

A FastAPI backend that embeds the company PDFs, stores the vectors, and retrieves the relevant chunks per question.
Gemini flash to turn those chunks plus the question into a short, spoken-style answer.
The browser for the voice on both ends: webkitSpeechRecognition for speech-to-text, speechSynthesis for text-to-speech. Free, no third-party voice API.

Text mode worked on the first try. You type a question, it retrieves, Gemini answers. Then I wired up the microphone, and things stopped being simple.

Problem 1: The Microphone That Refused to Listen

The symptom was maddeningly vague. The mic button would turn green like it was listening, I'd talk, and nothing would happen. No transcript, no answer, no error dialog. The text-to-speech side worked, so the assistant could greet me out loud, but it never heard a word I said back.

The trap here is that webkitSpeechRecognition looks like a local browser feature. It isn't. Under the hood it streams your audio to a Google speech service and streams text back. That means it depends on a network endpoint that can be blocked, rate-limited, or simply turned off by the browser vendor. Brave, for instance, disables it on purpose for privacy reasons, because it doesn't want to route your microphone through Google. And on the network I was testing from, even Chromium browsers couldn't reach the service.

Chasing It With an On-Screen Debug Log

I couldn't see the browser console easily on the device I was testing, so I did the unglamorous thing and printed a live event log right onto the page: every recognition start, result, error, and end, timestamped. That turned a vague "it doesn't work" into something precise.

[12:08:32] mic error: network
[12:08:32] mic ended
[12:08:31] mic started (listening)
[12:08:31] mic error: network
[12:08:30] mic started (listening)
[12:08:30] mic error: network
...

There it was, over and over: network. The recognition object was starting fine, getting the microphone, and then failing the instant it tried to reach the speech backend. My auto-restart loop was faithfully retrying and failing every single time. No amount of frontend cleverness was going to fix a blocked network call. The browser's speech recognition was a dead end in this environment.

The Fix: Move Speech-to-Text to the Server

The right move was to stop treating the browser as a transcription service and treat it as what it actually is: a good audio recorder. So I flipped the architecture. The browser records raw audio, and my backend does the transcription with Gemini, which accepts audio natively. That works on any browser and any network, and it happens to be the more production-correct approach anyway.

On the frontend I dropped webkitSpeechRecognition entirely and used the Web Audio API to capture PCM samples. Two small details mattered. First, I watched the audio's RMS level to detect when the user had actually started speaking and then paused, so recording stops automatically after about a second of silence instead of needing a stop button. Second, I encoded the captured samples to a 16 kHz mono WAV in the browser, because WAV is a format Gemini reliably accepts, unlike the WebM/Opus that MediaRecorder hands you by default.

// stop automatically once the user finishes a sentence
if (rms > SILENCE_THRESHOLD) { speechStarted = true; silenceStart = 0; }
else if (speechStarted) {
  if (!silenceStart) silenceStart = now;
  else if (now - silenceStart > 1000) stopRecording();  // 1s of quiet
}

The backend endpoint then just handed the audio bytes to Gemini and asked for a transcription:

resp = client.models.generate_content(
    model=CHAT_MODEL,
    contents=[
        types.Part.from_bytes(data=audio_bytes, mime_type="audio/wav"),
        "Transcribe this audio to plain text. Return only the words spoken.",
    ],
)
return resp.text.strip()

Record in the browser, transcribe on the server, run RAG, answer, and speak the reply back with the browser's text-to-speech (which was never the problem). The moment I stopped depending on the browser's hidden speech service, the whole thing worked, in Chrome, Edge, and even Brave.

Problem 2: Answers That Cut Off Mid-Sentence

With voice working, a second bug showed up. Replies kept getting truncated. "An expense ratio is simply the annual fee that a mutual" and then nothing. I'd set a max_output_tokens limit, so my first instinct was that the answers were just too long. They weren't.

The Gemini flash model I was using is a "thinking" model. It spends output tokens on internal reasoning before it writes the visible answer, and that reasoning is billed against the same max_output_tokens budget. A quick look at the token usage was the giveaway: on one call, 429 tokens went to "thoughts" and only 12 to the actual reply. My cap was being eaten by invisible thinking before the answer had a chance to finish.

For a voice bot that just needs short, friendly answers, that reasoning is wasted latency and wasted tokens. The fix was to turn it off, which the newer SDK exposes directly:

config=types.GenerateContentConfig(
    system_instruction=SYSTEM_PROMPT,
    max_output_tokens=400,
    thinking_config=types.ThinkingConfig(thinking_budget=0),  # no hidden thinking
)

Two problems solved with one change. Answers stopped truncating because the whole budget now goes to the visible reply, and responses got roughly two to three times faster because the model stopped pausing to think about a question that didn't need it.

Problem 3: The Phantom 405

One more that cost me twenty confused minutes. After adding the new /transcribe endpoint, calls to it kept returning 405 Method Not Allowed, as if the route didn't exist, even though it was clearly defined and the file compiled. The route hadn't disappeared. I had two uvicorn processes bound to the same port from earlier restarts, and the stale one, running the old code without the new route, was answering some of the requests. The old server fell through to the static-file handler, which only allows GET, hence the 405.

The fix was boring: kill every stale process, confirm only one was listening on the port, and start a single clean server. The lesson that stuck was to check what's actually bound to the port before debugging a "route not found" that makes no sense, because a zombie dev server will happily lie to you.

What I Took From It

The interesting part of this project wasn't the RAG or the model. It was that the pieces which looked free and local, the browser's built-in speech APIs, were the least reliable, because one of them quietly depends on a remote service you don't control. The debug log on the page is what turned that from a mystery into a one-word diagnosis, and moving transcription to the server turned a browser-specific hack into something that just works everywhere.

None of the individual fixes are clever. Record audio instead of trusting the browser to transcribe it. Turn off thinking when you don't need it. Don't run two servers on one port. But stringing them together is what took a demo that only worked in text into a voice assistant that actually holds a conversation.

← Back to portfolio