A note on status: this is an early write-up of a system that works but is not yet production-ready. I am documenting the design and the decisions behind it while I finish hardening it. A polished version, along with the security and performance work described at the end, will follow in the coming days. Treat this as a design log rather than a final report.
Motivation
Hosted voice APIs are expensive. A complete voice interaction requires three billed stages, speech-to-text, a language model, and text-to-speech, and the combined per-minute and per-token costs make experimentation costly well before a project reaches any scale. For prototyping and low-traffic workloads, that pricing is difficult to justify.
I set a deliberate constraint for this project: build a fully functional voice assistant with a recurring infrastructure cost of zero. Not low-cost, zero. Free-tier services only. The constraint was partly practical and partly an engineering exercise: it forces you to understand each component well enough to assemble it yourself rather than paying to have the integration hidden.
This document describes the resulting system, the architectural decisions, and the operational problems encountered along the way.
System Overview
The system is a three-stage pipeline. Audio captured in the browser is processed sequentially:
- Speech-to-Text (STT), transcribe the user's speech to text.
- Language model (LLM), generate a conversational reply.
- Text-to-Speech (TTS), synthesize the reply back to audio.
Each stage is served by a provider with a viable free tier:
- STT, Cloudflare Workers AI (Whisper), executed at the edge.
- LLM, Groq as the primary provider, with Cloudflare Workers AI (Llama 3.1 8B) as an automatic fallback.
- TTS, Piper, an open-source neural TTS engine, self-hosted on free compute (detailed below).
A single Cloudflare Worker orchestrates the pipeline and also serves the client, so there is no separately hosted frontend to maintain.
Architecture
Browser widget --WebSocket--> Cloudflare Worker --HTTP--> Oracle Cloud VMs
(mic + speaker) (Durable Objects) (Piper TTS)
|
1. Whisper STT (Workers AI)
2. LLM (Groq -> Llama fallback)
3. Piper TTS (one of two free VMs)
Client
The client is a single self-contained HTML page served directly from the Worker. It records audio with the MediaRecorder API, streams it over a WebSocket, and plays back the synthesized response. There is intentionally no frontend framework or build step; the surface area is small enough not to warrant one.
Orchestration Layer
The Cloudflare Worker is the control plane. It serves the client at the root path,
opens a per-connection voice session at /ws, and exposes
/health and /scheduler/status endpoints for monitoring.
Each WebSocket connection is backed by a Durable Object, a single-threaded, stateful execution context scoped to that session, which runs the full transcribe-reason-synthesize pipeline. The session rejects incoming audio while a previous request is still in flight, which keeps concurrent clips from interleaving within a session.
Language Model with Fallback
Groq serves as the primary LLM provider on latency and free-tier grounds. To avoid a single point of failure, any error from the Groq call, rate limiting, an outage, a malformed response, triggers a transparent fallback to Cloudflare Workers AI (Llama 3.1 8B). The fallback is invisible to the user and keeps the assistant responsive when the primary provider is unavailable.
Self-Hosted TTS on Free Compute
Text-to-speech is the stage I chose to self-host, because hosted TTS was the largest recurring cost in the providers I evaluated. Piper runs on Oracle Cloud's Always-Free tier, which provides persistent Linux VMs at no charge. The instances are modest, VM.Standard.E2.1.Micro, one OCPU and one gigabyte of memory, but they are full machines with public IPs.
I provisioned two of them, for two distinct reasons:
- Capacity. The instances are resource-constrained. A single VM occupied with one synthesis request should not block other sessions. Two instances provide two independent synthesis slots.
- Redundancy. If one instance fails, crashes, or becomes unreachable, the second continues to serve requests. No single machine failure takes the TTS layer offline.
Each VM runs Piper with two voice models, Indian English as the default and Hindi as
a secondary, wrapped in a small FastAPI service that exposes a /synthesize
endpoint. The service runs under systemd on port 8080, so it restarts on failure and
survives reboots.
Slot Scheduling
With two backends and concurrent sessions, the system must avoid dispatching two synthesis jobs to the same instance simultaneously. This is handled by a single global Durable Object that acts as a scheduler. A session claims a free slot before synthesizing and releases it on completion; if both slots are occupied, the request waits and retries. A reset endpoint is provided to clear a slot that becomes stuck, for example, if an instance crashes mid-request and never releases its slot.
The result is one Worker coordinating two free Oracle instances, with one effectively serving as a hot standby, at no recurring cost.
Operational Issues and Resolutions
The system did not work on the first attempt. The most instructive problems are documented below.
Two-Layer Network Filtering
Both instances became unreachable on port 8080 while Piper continued to respond correctly on localhost. The cause was that Oracle enforces filtering at two independent layers: the VCN Security List (a cloud-level network firewall configured in the console) and the instance's own iptables configuration. Port 8080 must be opened in both. Opening only one produces a misleading symptom, the service is healthy locally, but external requests time out.
iptables Rule Ordering
After adding the iptables rule, one instance still refused connections. The rule was present but had been inserted below the chain's blanket REJECT rule. iptables evaluates rules top to bottom and stops at the first match, so the allow rule for 8080 was never reached. The fix was to reposition it above the REJECT rule. This is a routine mistake, but it is easy to miss because the rule appears correct in isolation.
Synthesis Latency on Constrained Hardware
The free micro instances are slow under TTS load. A short response synthesizes in roughly six seconds; a longer three-sentence response can take thirty seconds or more. This is a hardware constraint rather than a defect, and it is the principal trade-off of running on free compute. Mitigations under consideration are shorter generated responses and a lighter voice model.
Failover Without Penalizing Slow Synthesis
The latency above complicates timeout handling. A short request timeout would truncate legitimate but slow synthesis; a long one would cause a dead instance to stall a request for the full duration before failing over. The chosen approach separates liveness from work: a fast health check (a four-second probe) detects an unreachable instance and fails over immediately, and only after an instance proves responsive is it granted a generous window to complete the slower synthesis. The system fails fast on unhealthy backends while remaining patient with healthy but slow ones.
Known Limitations
This is a working prototype, and several areas are not yet production-grade:
- Synthesis latency on the free instances is significant, as described above.
- Hindi support is wired through the full pipeline, but there is no language detection yet, so responses are currently always synthesized in English.
- The interaction model is single-turn push-to-talk. There is no continuous conversation, interruption handling, or cross-turn memory.
- Error handling is functional but minimal.
Planned Work
Before publishing a finalized version, I intend to:
- Reduce TTS latency through shorter responses and a lighter voice model.
- Add automatic language detection to select between Hindi and English.
- Evaluate a continuous-conversation interaction model in place of push-to-talk.
Conclusion
The objective was to determine whether a complete, usable voice assistant could be built with no recurring infrastructure cost. It can. The system delivers a full speech-to-text, language-model, and text-to-speech pipeline entirely on free-tier services, backed by a self-hosted, distributed TTS layer across two free Oracle instances with automatic failover.
The implementation has clear limitations, notably synthesis latency and the outstanding security work, but it functions, it costs nothing to operate, and the process clarified a number of practical details about edge compute, network filtering, and orchestration that are easy to take for granted when they are handled by a paid abstraction.
The system is being hardened now, and a finalized version will follow shortly.