Voice AI & Telephony

Architecting a Coordinated Multi-Agent Voice Runtime for Enterprise Customer Service

A proposed real-time voice AI system where six specialist agents - research, workflow, policy, tools, and verification - work beneath a single natural conversation, coordinated by a supervisor, so callers get resolved requests instead of a script.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

CS-025_Multi_Agent_Voice_Runtime image 1

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client's customer service operation needed more than a question-and-answer bot bolted onto a phone line. Callers needed a live, interruptible conversation that could actually resolve a request - check an account, reverse a charge, update a record - not just describe what a human agent would eventually do.

Pfactorial Technologies proposed a production-grade voice AI runtime built on LiveKit Agents, a real-time voice framework purpose-built for natural interruption handling and low-latency streaming speech. Underneath a single continuous call, a coordinated team of six specialist agents works in parallel: one holds the conversation, one researches in the background, one keeps the interaction on a structured path, one checks policy, one executes actions against real business systems, and one verifies the answer before it is ever spoken.

The design is explicit about what distinguishes this from a simpler two-model pipeline. Each agent owns one narrow responsibility with its own judgment, a supervisor coordinates the six, and every decision is logged on its own - so a new tool, policy, or workflow can be added later to one agent without retraining or rewriting the conversation itself.

Why this engagement is representative This engagement demonstrates Pfactorial's approach to voice AI for regulated, action-taking workflows: separating conversation from judgment so that policy, verification, and tool execution are each owned by a distinct, testable, traceable agent rather than folded into one model's best guess.

THE CHALLENGE

The client's existing options were either a rigid IVR menu tree or a chatbot that could describe an action but not safely perform one. Four problems shaped the proposed design.

1. Callers need resolution, not just answers

A basic question-and-answer bot can describe what should happen, but cannot safely execute an action - a reversal, an update, a booking - against a live business system without validation, retries, and an audit trail.

2. One model doing everything is not a multi-agent system

Running a second model to research in the background while a first model holds the conversation is a useful two-model pipeline, but quality issues stay untraceable: whichever model happens to weigh a signal more heavily in the moment wins, with no clear owner for a wrong answer.

3. Actions on real systems have to be safe to retry

A tool call that fails or times out mid-call cannot be silently retried without risk of duplicating a booking or a charge; failures need to be handled explicitly, not hidden from the caller.

4. Escalation must never be a dead end

Callers who ask for a person, hit a policy boundary, or reach an ambiguous outcome need a handoff that carries full context - not a restart, and not a bot that keeps guessing.

The real brief Not "build a chatbot that sounds helpful" but "build an assistant that resolves real requests, checks policy before anything happens, and leaves a complete, reviewable trace of what was said and done."

THE SOLUTION

Pfactorial proposed organizing the runtime around a supervisor and six specialist agents, each with a single, narrow responsibility, rather than one model asked to converse, research, decide, and verify all at once.

CS-025_Multi_Agent_Voice_Runtime image 2

Figure 1 - Six specialist agents coordinated by a supervisor, from a live call to a policy-checked, verified resolution.

Architectural principles

One responsibility per agent - the Conversation, Research, Workflow, Policy, Tool Gateway, and Response Verifier agents each own a single job, so a quality issue can be traced to the agent responsible rather than guessed at across one monolithic model.
Built on a real-time voice runtime, not adapted from text - LiveKit Agents was selected specifically for natural interruption handling and low-latency streaming speech-to-text and text-to-speech - a foundation purpose-built for live voice, not a text chatbot repurposed for a phone line.
Policy checked twice, not assumed - the Policy Engine is consulted once to scope what the conversation may even discuss, and again to approve the concrete action - a second check that cannot be worn down by a long, meandering conversation because it evaluates the action, not the dialogue.
Fails safe, escalates rather than guesses - when a tool call fails, research turns up nothing, two systems disagree, or a pattern suggests misuse, the interaction is held or escalated by design rather than allowed to continue on uncertain footing.

CAPABILITIES DELIVERED

The proposed runtime's capabilities span the full path from a live call to a resolved, auditable outcome.

CAPABILITY	WHAT IT DOES
Real-Time, Interruptible Conversation	Callers can talk over the assistant and be understood mid-sentence, the way they would with a person, rather than waiting for a turn.
Parallel Background Research	A dedicated Research Agent pulls live information from documents and business systems while the conversation keeps moving, without pausing the call.
Guided Structured Workflows	A Workflow Engine keeps each interaction on a path built to actually resolve the caller's request, rather than a free-form exchange that may or may not land anywhere.
Policy-Gated Tool Execution	A Tool Gateway executes actions against real systems only after policy approval, with validation and safe, non-duplicating retries.
Response Verification	Nothing is spoken to the caller until the Response Verifier has checked it against what the system actually did, not what it was merely supposed to do.
Human Handoff & Full Tracing	An escalation path is always available, and every conversation produces a complete, reviewable trace of what was said and done - usable for review and improvement after the fact.

CS-025_Multi_Agent_Voice_Runtime image 3

Figure 2 - A typical interaction: how a call moves from issue to a verified resolution, with an escalation path available at every uncertain step.

Design note The first build is deliberately scoped to prove the full pattern end to end on one supported workflow, with five to ten validated tools, rather than spreading effort across many workflows at once. Additional workflows, tools, and channels are designed to extend the same core runtime later, without a redesign.

ENGINEERING FOR SCALE AND RELIABILITY

Several engineering decisions in the proposed design distinguish a runtime meant to hold up on real, regulated business workflows from a demo that only works on a scripted call.

A real-time runtime chosen for the workload

LiveKit Agents was selected specifically for natural interruption handling and low-latency streaming speech, giving a foundation that scales from a single supported workflow to many without redesigning the conversation itself.

Coordination via agent handoffs, not a single prompt

the supervisor routes what the caller says among specialist agents and enforces that policy is checked before anything sensitive happens, rather than relying on one model to juggle every responsibility inside a single context window.

Durable workflows and safe retries

Temporal is proposed to keep structured workflows and tool retries durable even if a step fails mid-call, so a network timeout cannot silently double-book an appointment or duplicate a charge.

Two-check policy gating

the Policy Engine is consulted once to scope what the conversation may discuss and again to approve the concrete action, so approval logic cannot be gradually talked around over the course of a long call.

Tracing built in, not bolted on

OpenTelemetry is proposed for full tracing of latency, cost, and quality across every conversation from the first build, rather than added retroactively once the system is already in production.

A lean stack, each piece earning its place

FastAPI, Redis, and PostgreSQL were selected specifically for session state, durable records, and transcripts at the scope of the first build, rather than adding infrastructure for its own sake ahead of proven need.

DELIVERY APPROACH

The proposed first build is scoped to prove the full multi-agent pattern end to end on one workflow across four phases, moving from kickoff to a fully traced, testable voice assistant in roughly eight weeks.

1. Foundation (Weeks 1-2, 78 hrs) - live voice session setup across phone and browser entry points, streaming speech-to-text and text-to-speech integration, turn detection and natural interruption handling, and session state management.

2. Intelligence (Weeks 3-4, 100 hrs) - a parallel Research Agent for background lookups, one complete structured customer-service workflow, and a Tool Gateway covering five to ten validated tools with retry handling.

3. Assurance (Weeks 5-6, 88 hrs) - the Policy Engine's approval rules and action gating logic, the Response Verifier's accuracy check before anything is spoken, human handoff flow and escalation triggers, and full event tracing across the interaction.

4. Validation (Weeks 7-8, 66 hrs) - an automated test suite covering the workflow end to end, structured review and refinement, and documentation and handover.

RESULTS AND IMPACT

CS-025_Multi_Agent_Voice_Runtime image 4

Figure - Key outcomes from this engagement.

The proposed engagement gives the client a concrete first-build plan: a six-agent architecture with defined responsibilities, a four-phase delivery path with hours estimated per phase, and explicit success criteria - callers complete the supported workflow without repeating themselves, interruptions are handled naturally, no action reaches a real system without a policy check, and every conversation produces a complete trace.

By scoping the first build to one workflow, proven end to end with five to ten validated tools, the design is structured so the pattern - not just the specific workflow - is what gets proven first, with additional workflows, tools, and channels extending the same runtime afterward rather than requiring it to be rebuilt.

What it enabled commercially

If built as proposed, the runtime is designed to give the client faster resolutions, fewer repeat calls, actions that are checked before they happen, and a complete record of what was said and done on every call - without adding headcount to the contact center.

WHY PFACTORIAL

This engagement draws on Pfactorial's AI product engineering capability: designing coordinated multi-agent voice systems where conversation, research, policy, and action are owned by separate, testable agents rather than left to a single model's best guess.

CS-025_Multi_Agent_Voice_Runtime image 5

Figure - Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with customer service and contact center teams evaluating whether a voice AI system can safely take real action, not just answer questions. If you are weighing a multi-agent voice architecture against a simpler bot, we are happy to give you an honest read on scope, cost, and risk before anyone commits to anything. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.