Skip to main content
Target: 10 concurrent voice sessions, 8K LLM context window. STT and TTS model weights are shared across sessions; LLM KV cache is per-session and dominates VRAM. All VRAM figures are indicative and depend on quantization and serving configuration.

​
LLM VRAM (self-hosted catalog)

​
STT VRAM (shared weights + per-session)

The emotion2vec paralinguistic sidecar (emotion + sentiment) is lightweight and runs alongside a transcribing engine on CPU or GPU.

​
TTS VRAM (shared weights + per-session)

​
GPU tier matrix

Minimum: fits the stated workload at 4K context, no headroom. Stable: 8K context with ~30 % VRAM headroom for fragmentation + burst concurrency. Self-hosted inference images are built per accelerator — CPU, CUDA, and ROCm variants — so the same catalog runs on NVIDIA and AMD hardware, with CPU fallbacks for the lightweight engines.

​
Host sizing per GPU node

​
Sizing rules

Concurrency is KV-cache bound. Halving context length halves per-session memory cost.
LLM throughput is memory-bandwidth bound, not FLOPs — HBM3/HBM3e matters more than core count.
The self-hosted LLM plane runs vLLM: paged KV cache + continuous batching gives roughly 3× the concurrency of a vanilla Ollama setup on the same hardware.

​
Topology options

​
Single host (dedicated instance)

The full platform on one Linux host as a self-contained container bundle — the standard shape for dedicated and on-prem instances. Suitable for sub-10-concurrent voice loads on a single GPU box (or CPU-only when using cloud inference providers). Ships with a guided update script and restore runbook.

​
Kubernetes cluster

The same services deployed from the platform’s Kubernetes charts, for customers operating their own clusters or larger multi-node footprints.

​
Split (control plane + GPU)

Control-plane services (frontend, backend, database, media, cache) on CPU hosts; the GPU-bound services (STT, TTS, LLM, agent worker) on one or more GPU hosts. Connect the two over a private network — VPN, WireGuard, Tailscale, or private VPC peering. For larger fleets, the split scales to many GPU hosts and a single control plane.