# Voice Keyboard: Self-Hosted Whisper Dictation That Types into Your Apps Author: AlgoFly Team Published: 2026-09-25T11:23:23+00:00 Canonical: https://algofly.ai/blog/voice-keyboard Dictation becomes much more useful when the words land where you are already working. A browser transcript helps, but copying it into an editor, chat window or document after every recording interrupts the flow. [Voice Keyboard](https://github.com/algofly-oss/voice-keyboard) connects a browser microphone, a self\-hosted Whisper server and a selected keyboard receiver. A phone can capture speech while a workstation receives the words. The interesting engineering is between those endpoints: deciding when speech is ready for recognition, preserving sentence continuity across cuts, reconnecting without immediately abandoning a recording, and translating text into input that the destination operating system actually accepts. The same interface also provides a keyboard and touchpad, so reaching for another device does not have to end the workflow. ## Audio streams continuously; recognition happens in utterances There are two different meanings of streaming here. Microphone audio travels continuously to the gateway over a WebSocket. Recognition operates on completed audio pieces: Silero voice activity detection finds useful boundaries, and the gateway submits each piece to an ordinary Whisper transcription request while recording continues. Text can therefore reach the selected client before the entire recording ends. This implementation does not expose Whisper decoder tokens as they are generated. It does not repeatedly decode overlapping windows and commit a stable word prefix. Its public stream messages are `segment` and `final`: a segment contains a recognized utterance with its separator, while final contains the assembled text. That distinction explains both the responsiveness and the tradeoff: shorter utterances arrive sooner, but longer ones give recognition more context. ``` Browser microphone → AudioWorklet → binary PCM WebSocket ↓ Silero VAD → utterance queue ↓ shared Whisper inference slot ↓ sentence-aware joiner → selected receiver ↓ focused application ``` The browser, model server and receiver each do a different job. The receiving laptop does not need to run Whisper, and the microphone device does not need a desktop typing backend. This separation makes the same speech pipeline useful for a powerful workstation, a small always\-on server, or an optional Bluetooth keyboard board. ## From browser audio to predictable PCM frames The browser requests a mono microphone stream with echo cancellation, noise suppression and automatic gain control. An AudioWorklet reads the samples on the audio processing path rather than relying on a UI timer. It averages incoming samples into a 16 kHz output stream, clamps their amplitude, and converts them to signed 16\-bit values. The worklet collects 1,600 samples before transferring an ArrayBuffer to the page. At 16 kHz that is about 100 milliseconds of audio, or 3,200 bytes. These are transport chunks, not speech boundaries. The gateway interprets their binary payload as little\-endian int16 PCM; a word can span several frames without being split into separate recognition requests. Stopping recording asks the worklet to flush its incomplete final buffer before the page sends a stop message. A 700\-millisecond fallback prevents a stalled audio thread from blocking that stop indefinitely. A separate watchdog attempts to resume a suspended audio context after missing chunks, then reports a microphone failure if audio still does not arrive. Reusing one AudioContext also handles the practical restrictions of iOS Safari, where newly created contexts can remain suspended. Microphone access still requires a secure browser context and permission. The included Caddy setup provides HTTPS; a local certificate authority must be trusted on the browser device. Native client certificate handling and browser trust are separate concerns. A page loading successfully is not proof that its microphone is allowed to start. ## Silero finds pauses without treating every dip as silence The gateway feeds Silero 512\-sample windows, corresponding to 32 milliseconds at the fixed sample rate. Its speech decision uses hysteresis: a probability of at least 0\.5 enters speech, while a probability below 0\.35 leaves it. Values between those thresholds preserve the previous state, reducing rapid switching during brief dips inside a word. Before speech starts, the segmenter keeps a short lead\-in rather than accumulating an entire room's silence. It also drops pieces with less than roughly 250 milliseconds of speech evidence. This is a practical filter for tiny fragments, not a guarantee that every non\-speech sound or hallucinated phrase is eliminated. Live typing exposes four pace policies. The default Fast policy waits for approximately 400 milliseconds of silence once at least one second of audio has accumulated, looks for a shorter gap after five seconds, and prevents a piece growing beyond eight seconds. Instant, Balanced and Accurate choose different limits. These are configured segmentation targets, quantized to VAD windows, rather than measured end\-to\-end latency promises. A speaker may continue without a clear pause. Once the hard limit is reached, the segmenter searches the final two seconds for the least speech\-like point and cuts there. It marks that boundary as forced, which matters later when joining text. The longer\-context non\-live policy uses larger pieces, including a 28\-second hard limit. Translation especially benefits from having more of a sentence available before inference. ## Recognize a piece, then repair the seam Each completed piece is wrapped as a mono WAV file and posted to the configured Whisper service. Transcription uses the transcription endpoint; translation uses the translation endpoint. The gateway fixes the model to the deployed model rather than allowing an arbitrary stream request to load another one. Within one dictation, a worker consumes pieces in order. Each request can include the user's vocabulary prompt plus the last 200 characters of raw recognized history. Keeping that history in Whisper's original script gives the next request useful spelling and language context, even when the displayed result will later be romanized. The Joiner handles a subtle consequence of chunking: Whisper often ends a piece with a full stop and capitalizes the next piece even when a forced cut interrupted one sentence. At a forced boundary, the joiner can hold back the full stop. When the next piece begins with a recognized continuation word such as “and” or “because,” it joins the text with appropriate casing; otherwise it can restore a sentence boundary. Leading or trailing ellipses from cut\-off phrases are handled separately. Segments already carry their leading space or punctuation, so receivers concatenate and type them in order. When recording stops, the segmenter flushes remaining audio, the worker drains its queue, and the joiner emits any held closing punctuation before final text is returned. This is append\-oriented sentence repair. It does not silently rewrite words already typed into another application. ``` {"type":"start","language":"en","live":true,"pace":"fast","client":"browser-example"} {"type":"segment","text":"Words from a completed utterance"} {"type":"segment","text":" and its continuation."} {"type":"final","text":"Words from a completed utterance and its continuation."} ``` This illustrates the actual message shapes. Binary audio frames travel between start and stop; the example text is explanatory, not a captured recognition result. ## Share the model, and make interruption behavior explicit Every account shares the configured Whisper model. A semaphore limits concurrent inference, with a default of one request at a time. This avoids assuming that a GPU can run every user's inference simultaneously. Increasing concurrency is a deployment choice that depends on model memory and hardware, not an automatic speed improvement. Audio ingestion and recognition are decoupled by an utterance queue, so recording can continue during inference. The current server queue is not bounded by an explicit maximum size. The semaphore controls inference concurrency; it is not a complete admission\-control or memory\-backpressure system. If sustained inference falls behind incoming speech, queued work and finishing time can grow. Tune the model and pace against realistic recordings and concurrent use. The browser also handles a dropped stream. It continues recording into a pending buffer and retries with exponential backoff from one to ten seconds. The pending buffer retains the newest five minutes of audio. A separate replay log retains up to 30 seconds of recently sent audio; after a recognized segment arrives, older entries are trimmed while keeping roughly the last 1\.5 seconds. That replay is a recovery heuristic for audio the dead connection may not have processed. It is not an exactly\-once transcription protocol with acknowledged audio offsets, and it cannot promise that every boundary survives every failure without repetition or loss. The stop path has a finishing timeout and can report an untranscribed ending instead of claiming success indefinitely. Receiver delivery has different semantics. Keyboard messages can wait briefly when the selected client disconnects: the room retains at most 500 pending messages and only replays those younger than 60 seconds. Pointer movement is never queued. Replaying an old sentence may be useful after a brief reconnect; replaying a stale drag or click could move the wrong thing. ## One account room, one receiving client A Room holds one account's web watchers, device connections, selected receiver and active dictation. Rooms are keyed by account ID. Several browsers can observe that account's recording and receive level or segment events, but a second simultaneous dictation in the same room is rejected to prevent overlapping input. The selected device is an account choice, not a broadcast to every connected computer. Selection changes notify clients and browser interfaces. This lets a phone remain the control surface while the user switches the destination between machines. Focus still belongs to the destination operating system: place the cursor in a scratch document before the first test. Enrollment exchanges an account installation token for an individual device credential. Device credentials are stored as SHA\-256 hashes, while account passwords use salted scrypt hashing. Browser sessions are signed and tied to an account session epoch, allowing password changes to invalidate prior sessions. These mechanisms describe the implementation; they do not replace careful credential storage, HTTPS or a deployment security review. ## Turn text into native input, with real platform limits The desktop receiver is a self\-contained Go program for supported Windows, macOS and Linux builds. It maintains a WebSocket connection, receives ordered text and key messages, and invokes the platform's input backend. A phone therefore does not need an editor\-specific plugin on every application it might target. macOS uses CoreGraphics Unicode events and requires Accessibility permission. Windows uses SendInput Unicode events; a non\-elevated client cannot type into an elevated application. Linux X11 and Wayland take different paths: the documented Wayland uinput method needs a one\-time permission setup and is limited to US\-layout ASCII. Complete Unicode parity across platforms would be an inaccurate promise. Supported macOS and Windows paste modes can help with longer text while restoring supported previous clipboard content. The receiver also offers typing\-speed settings. These choices matter because a program that accepts ordinary typing quickly may react differently to a large clipboard insertion or simulated shortcut. The ESP32 option replaces the desktop agent with a Wi\-Fi\-connected Bluetooth HID receiver. Supported ESP32 and ESP32\-C3 firmware receives server messages and presents a keyboard to its paired host. Its keymap translates characters into US\-layout key strokes, normalizes supported accented Latin characters and skips unsupported scripts. It needs server connectivity, power and Bluetooth pairing; it is not a speech model running on the microcontroller. ![Voice Keyboard browser interface with its touchpad, on-screen keyboard and microphone control](/content/product-stories/voice-keyboard/overview.webp) The actual browser control surface combines a touchpad, keyboard and microphone control. This idle view does not represent a transcription benchmark. ## Make the microphone device a useful control surface Dictation is only part of entering text. The browser includes an on\-screen keyboard, named keys and held\-key states, with an optional F1–F12 row. A system\-keyboard path forwards ordinary characters and supported named keys. This helps the user navigate, correct a word, submit a field or continue with manual input without changing devices. The touchpad translates relative movement, scrolling, clicks and button press/release into checked pointer messages. The gateway validates action names and buttons and clamps movement deltas before forwarding them to the selected receiver. One\-finger movement, two\-finger scrolling and dedicated mouse buttons support practical navigation and dragging; the actual input backend still determines platform behavior. Layout responds to the available space. A tablet in landscape can keep both controls visible, while a short landscape phone can switch between keyboard and touchpad views. An ESP32 Bluetooth keyboard can also be temporarily disconnected through the interface so a paired tablet shows its own on\-screen keyboard again; requesting input reconnects it. These details solve the less glamorous problem of keeping dictation usable after the first sentence. ## Choose language, model and an upgrade path deliberately The default transcription model is large\-v3\-turbo; the translation variant uses large\-v3 for English output. Hinglish selects Hindi recognition and then converts recognized Indic text into Latin letters. The romanizer applies script mappings and spelling rules, including schwa deletion. This is post\-processing, not a separate speech inference pass or a promise of arbitrary\-language translation. Receiver script limits still apply. Start with the public repository's setup script: ``` git clone https://github.com/algofly-oss/voice-keyboard.git cd voice-keyboard ./setup.sh # Check service state and model initialization: docker compose ps docker compose logs -f transcription ``` CPU operation uses the corresponding CPU service and is slower. Initial model downloads and server resources are required. Configure trusted HTTPS before diagnosing microphone problems, then verify model readiness, receiver selection and operating\-system permissions in that order. Clients learn the deployment's alternative addresses and instance identity. Address ranking prefers reachable local routes while checking that alternatives belong to the same server. This helps when moving between LAN and remote access, but browser certificate trust and sign\-in still apply per address. Desktop self\-updates download from the configured server's client path, verify the advertised size and SHA\-256, run the candidate's version command, then replace the executable and restart. Windows uses an old/new rename sequence because its running executable cannot be replaced like a Unix file. A failed automatic version waits before retrying. These checks catch damaged or unsuitable downloads; a hash supplied by the same trusted server is not an independent release\-signing system. Read the [repository](https://github.com/algofly-oss/voice-keyboard), [desktop guide](https://github.com/algofly-oss/voice-keyboard/blob/main/clients/desktop/README.md), [ESP32 guide](https://github.com/algofly-oss/voice-keyboard/blob/main/clients/esp32/README.md) and [HTTPS scenarios](https://github.com/algofly-oss/voice-keyboard/blob/main/docs/https.md) for deployment details. The mechanism is intentionally understandable: continuous audio transport, phrase\-level recognition and a receiver you choose. Its behavior is governed by those concrete stages, rather than an unexplained “instant AI” claim.