Voice AI & Telephony

A Sub-Second, On-Premise Voice AI Platform for Telephony and Web

How Pfactorial Technologies engineered a real-time voice AI pipeline - speech-to-text, reasoning, and speech synthesis - tuned end to end for sub-second response on dedicated GPU infrastructure.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

Pfactorial_Case_Study_Low_Latency_Voice_AI_Platform image 1

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client needed a voice AI platform for telephony and web that felt genuinely conversational - not a system with the multi-second lag that makes AI voice agents feel obviously robotic and breaks natural turn-taking.

Every stage of a voice pipeline - transcription, reasoning, synthesis - adds latency, and stacking them naively produces a multi-second round trip that no amount of good conversation design can hide. Getting to sub-second response required optimizing each stage individually and the infrastructure underneath all of them.

Pfactorial built and performance-tested a custom voice AI deployment on dedicated GPU infrastructure (8x NVIDIA H200 and 8x RTX 5090), combining Silero VAD, Whisper transcription, a fine-tuned Llama model served via vLLM, and streaming ElevenLabs TTS - with barge-in interruption handling throughout.

Why this engagement is representative This engagement demonstrates Pfactorial's ability to work at the infrastructure and inference-optimization layer - GPU memory management, model serving, streaming audio - not just the conversational design layer most voice AI vendors stop at.

THE CHALLENGE

Reaching sub-second, natural-feeling voice interaction required solving latency and concurrency problems at every stage of the pipeline.

1. Every pipeline stage adds latency that compounds

Speech-to-text, LLM reasoning, and text-to-speech each take real time; stacked naively, they produce a multi-second delay that breaks the feeling of a real conversation.

2. Natural conversation requires handling interruptions

Real callers talk over the AI mid-response; without active interruption handling, the system either talks over the user or takes a beat to recover, both of which feel unnatural.

3. Concurrency and latency compete for the same GPU memory

As active sessions increase, KV cache and conversation state consume more GPU memory, and that memory pressure becomes the binding constraint on how many simultaneous calls the system can handle.

4. More hardware isn't always the right fix

Adding GPUs and CPUs would have solved concurrency limits, but at a cost that didn't need to be paid if the existing hardware could be used more efficiently.

The real brief Not “connect an STT, LLM and TTS API together” but “optimize every stage and the infrastructure underneath it until the pipeline feels like a real conversation, not a robotic relay.”

THE SOLUTION

Pfactorial optimized each pipeline stage individually - VAD-driven endpointing, GPU-served LLM inference with memory-efficient attention, and streaming speech synthesis - while tuning the infrastructure to get more out of existing hardware rather than defaulting to adding more.

Pfactorial_Case_Study_Low_Latency_Voice_AI_Platform image 2

Figure 2 - Sub-second, stage by stage: speech in, reasoning, speech out.

Architectural principles

Endpoint detection before transcription - Silero VAD detects speech boundaries and filters silence before Whisper runs, avoiding unnecessary inference on dead air and enabling faster turn detection.
Memory-efficient LLM serving - vLLM's continuous batching and PagedAttention manage KV cache memory efficiently, supporting roughly 50 concurrent sessions without excessive memory consumption.
Stream, don't wait for completion - Text-to-speech audio streams as it's generated rather than waiting for the full response, letting playback start earlier and reducing perceived response time.
Interruption as a first-class conversation state - Voice activity detection runs continuously, even during TTS playback - new speech immediately stops playback and redirects to the STT pipeline.

CAPABILITIES DELIVERED

Each capability targets a specific stage of the latency budget or a specific real-conversation behavior.

CAPABILITY	WHAT IT DOES
Real-time speech-to-text	Whisper Base with Silero VAD-driven endpointing, streaming audio processing.
Conversational AI orchestration	Fine-tuned Llama 1B Instruct served via vLLM with custom response guardrails.
Streaming text-to-speech	ElevenLabs voice synthesis, audio streamed as it's generated.
Barge-in interruption handling	Immediate playback cancellation and redirect to STT when the user starts speaking.
GPU-accelerated inference	H200 and RTX 5090 infrastructure with NVLink interconnect for the LLM layer.
Telephony/WebRTC integration	Supports both phone-based and web-based voice channels.

Pfactorial_Case_Study_Low_Latency_Voice_AI_Platform image 3

Figure 3 - The inference optimizations that made the concurrency and latency budget work together.

Design note The system was capable of sub-second responses under ideal conditions, but production latency typically landed at 900-1,200ms - the documentation is explicit about this gap between best-case and typical performance rather than only reporting the best number.

ENGINEERING FOR SCALE AND RELIABILITY

Six decisions carried the pipeline from a demo that works once to infrastructure that holds up under concurrent load.

Whisper Base chosen for the speed/quality balance

Rather than the largest available transcription model, Whisper Base was selected specifically for its balance between transcription quality and inference speed within the latency budget.

Continuous batching across sessions

vLLM's continuous batching processes requests from multiple concurrent voice sessions together, improving GPU utilization without introducing noticeable per-session latency.

Flash Attention for faster token generation

Flash Attention reduces memory access overhead in attention computation, directly improving Time-to-First-Token and keeping response times consistent during active conversations.

Geographic infrastructure placement

GPU infrastructure was deployed in data centers geographically close to the target business region specifically to minimize network round-trip latency.

Iterative, token-efficient prompt design

The system prompt was refined to remove redundant instructions, with frequently repeated behavioral rules moved into fine-tuning instead of being repeated in every prompt - reducing both prompt size and inference latency.

CPU-aware capacity planning

The voice pipeline was found not to be entirely GPU-bound - audio streaming, STT processing and session management all need CPU resources, so CPU capacity was tuned alongside GPU memory rather than treated as secondary.

DELIVERY APPROACH

The engagement built and performance-tuned each pipeline stage, then validated the whole system under realistic concurrent load.

1. STT layer build - Whisper Base with Silero VAD-based endpointing, tuned for streaming audio.

2. LLM layer deployment - fine-tuned Llama 1B Instruct served via vLLM with continuous batching and PagedAttention.

3. TTS layer evaluation and selection - comparing open-source options (Kokoro, VibeVoice) against ElevenLabs before selecting for production voice quality.

4. Interruption handling - continuous VAD monitoring and immediate playback cancellation on detected speech.

5. Prompt engineering & guardrails - structured prompts, hallucination mitigation, and token-usage optimization.

6. Concurrency and GPU-bottleneck tuning - profiling memory pressure at scale and tuning KV cache management rather than defaulting to more hardware.

RESULTS AND IMPACT

Pfactorial_Case_Study_Low_Latency_Voice_AI_Platform image 4

Figure 4 - Key outcomes from this engagement.

The platform achieved sub-second responses under ideal conditions and typical production latency of 900-1,200ms, supporting natural, low-lag voice conversations across telephony and web.

Roughly 50 concurrent voice sessions were supported on the existing GPU infrastructure through inference optimization, avoiding the cost of scaling hardware to meet concurrency demand.

What it enabled commercially

The client validated that a genuinely natural-feeling, low-latency voice AI experience was achievable on owned GPU infrastructure, with a clear picture of where the remaining concurrency bottlenecks are and what it would cost to scale past them.

WHY PFACTORIAL

This engagement reflects Pfactorial's ability to work below the conversational-design layer, at GPU infrastructure and inference-optimization depth, where most of the actual latency budget in a voice AI system is won or lost.

Pfactorial_Case_Study_Low_Latency_Voice_AI_Platform image 5

Figure 5 - Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with teams that need voice AI infrastructure engineered for real latency and concurrency requirements, not just a conversational demo. If you're evaluating a custom voice AI deployment, we're happy to give you an honest read on scope and risk before anyone commits to anything. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.

Pfactorial_Case_Study_Low_Latency_Voice_AI_Platform image 6

© 2026 Pfactorial Technologies. Client identity and product-specific implementation detail are withheld or generalized; no client data, credentials, source code, or infrastructure detail is included in this document.

Result and Analysis

ENGAGEMENT SNAPSHOT

How Pfactorial Technologies engineered a real-time voice AI pipeline - speech-to-text, reasoning, and speech synthesis - tuned end to end for sub-second response on dedicated GPU infrastructure.