RAG & Semantic Search

Designing Conversational and Voice Access Across a Scattered Document Estate

A proposed six-stage architecture that indexes documents where they already live - across Drive, OneDrive, SharePoint, and local files - and answers questions with a citation back to the exact source.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

CS-015_Document_Intelligence_Toolkit image 1

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client's documents lived across Google Drive, SharePoint, local machines, and meeting recordings, each with its own search that matched file names but understood nothing about how those files related to each other. Finding the signed version of a contract, or the number a quote's figures came from, meant opening several systems in sequence and guessing which of several similarly-named files was the real one.

Pfactorial Technologies proposed a six-stage system design that addresses two separate problems at once: search, so that related material scattered across systems can be found regardless of which system it sits in, and extraction, so that the facts locked inside prose - a payment term, a renewal date, a counterparty - become fields that can be filtered, sorted, and totaled rather than read one document at a time.

The design deliberately points at files rather than copying them: source systems remain the system of record, and the platform holds only an index - metadata, extracted text, embeddings, links, and extracted fields - so that clicking a result hands the user back to the native application with their existing permissions intact.

Why this engagement is representative This engagement demonstrates Pfactorial's approach to proposal-stage system design: naming the two genuinely different problems inside a document-intelligence request - search and extraction - and designing for each on its own terms, while stating plainly where a number can't yet be supported rather than asserting one that hasn't been proven.

THE CHALLENGE

Nothing about this system exists yet; the proposal responds to a real, everyday failure mode. Two separate difficulties needed different solutions.

1. Material is scattered across systems that don't talk to each other

Google Drive, SharePoint, and local files each maintain their own index, matching file names and at best some of the words inside. Nothing joins a contract to the email thread that negotiated it, the spreadsheet its numbers came from, or the meeting where it was agreed.

2. Documents are not databases

A contract contains a payment term, a renewal date, and a counterparty, but those facts are locked inside prose. No amount of better search turns them into something that can be filtered, sorted, or totaled - only reading the document and writing down what it says does that, which is exactly the manual work the system needs to replace.

3. File estates are mostly noise

A large share of any real file estate is screenshots, exported logs, template files nobody filled in, and copies of documents that were never used. Treating every file identically means paying repeatedly to be told there's nothing there, and cluttering every search result with it.

4. Different source types need entirely different handling

A clean PDF export, a photographed signed page, a Word document with tracked changes, an Excel grid, and an audio recording share almost nothing structurally - building for all five separately would mean building five systems rather than one.

The real brief Not "index everything" but "answer a specific question, with a citation back to the exact page it came from, regardless of which system the answer is actually sitting in."

THE SOLUTION

Pfactorial proposed a six-stage pipeline, with the first stage doing the specific job of flattening five very different source types into one common format before anything else happens.

CS-015_Document_Intelligence_Toolkit image 2

Figure 1 - Proposed six-stage architecture: collection & reading, storage, file-estate resolution, extraction, retrieval, and the agent that ties them together.

Architectural principles

Flatten variety before doing anything else - audio goes through speech-to-text and scanned pages go through OCR as the very first pipeline step, so every later stage only ever handles text and table structure - regardless of what the file originally was.
Three storage systems, each matched to its job - object storage for anything large or unsearched directly, a relational database as the system of record for everything filtered, sorted, or joined on, and a vector index purpose-built for meaning-based search - rather than one store stretched to do all three jobs.
Connect through shared anchors, not pairwise comparison - items are linked to a shared anchor (a client, a matter, a project) rather than compared against every other document directly, which avoids a cost that grows with the square of the corpus and produces a model that's easy to explain and audit.
Index and point, never copy - the system holds an index - metadata, extracted text, embeddings, links, and extracted fields - while files stay in their native systems, avoiding a second copy of the truth that would immediately begin to drift out of date.

CAPABILITIES DELIVERED

The proposed system's capabilities span the full journey of a document, from first being noticed to answering a question about it.

CAPABILITY	WHAT IT DOES
Multi-Source Collection	Ingests files from Drive, OneDrive, SharePoint, local folders via a desktop agent, and meeting recordings and transcripts.
Confidence-Routed File Handling	Routes every incoming item down one of three paths based on how confidently it can be placed, so noise never reaches the extraction or search layers.
Schema-Driven Extraction	A cheap classifier selects a document type, which selects a field schema; extraction returns nothing for a field it can't find rather than inventing a plausible value.
Dual-Path Retrieval	Structured field lookups and meaning/keyword search run in parallel, covering each other's blind spots on precision and completeness.
Permission-Filtered Search	Access is resolved and applied before any searching happens, so every downstream step only ever operates on material the requester is already allowed to see.
Cited, Agent-Composed Answers	An agent calls retrieval and other tools as needed, composing an answer where every claim links back to its source position for one-click verification.

CS-015_Document_Intelligence_Toolkit image 3

Figure 2 - How a question is answered: permission filtering first, dual-path search, related-material expansion, and a cited response.

Design note This document describes a proposed method, not a finished system. Where something is genuinely uncertain, or depends on the client's own document set, the proposal says so rather than asserting a number that can't be supported yet.

ENGINEERING FOR SCALE AND RELIABILITY

Several design decisions in the proposal distinguish a system that would hold up on a real, messy file estate from one that only works on a clean demo set.

Blocking narrows candidates before any expensive comparison

five proposed keys - folder/location, people involved, time, identifiers in the text, and semantic meaning - use cheap clues to eliminate obviously-wrong anchors before any costly comparison runs, so expensive matching only ever executes on a short list.

Keyword search is deliberately kept alongside meaning-based search

document work is full of exact tokens - contract numbers, clause references, part codes - that meaning-based search handles poorly because a number has no meaning to embed. Running both and combining rankings removes an entire category of failure at minimal cost.

Document text is data, never an instruction

permission checks live inside every tool rather than inside the agent, so text authored specifically to influence the model - a hidden instruction in a PDF footer - can never widen anyone's access, regardless of what it says.

Extraction schema is a business decision, not a technical default

the proposal deliberately avoids a general-purpose extractor that returns a hundred fields nobody uses, favoring an hour spent naming the twelve fields a specific team actually needs.

Low-confidence extractions go to a review queue, not into the database

a table of wrong figures poisons every report built on it; sending uncertain extractions to human review instead protects data quality and builds the labeled set future accuracy depends on.

The agent's clarification loop is deliberately capped

clarification is capped at one round and presented as clickable options rather than open-ended replies, so the system doesn't interrogate users, and a low-confidence answer says so plainly rather than inventing a plausible-sounding one.

DELIVERY APPROACH

As a proposal, the engagement's phases describe the proposed build sequence rather than a completed delivery history.

1. Collection and normalization - build readers for each source type, converting everything - including audio via speech-to-text and scans via OCR - into one common format before any downstream processing.

2. Storage layer - stand up the three-way split across object storage, the relational system of record, and the vector index, with a background resolver connecting related material.

3. File-estate resolution - implement the five-key blocking approach and the three-route confidence classification that decides how much processing each incoming item receives.

4. Schema-driven extraction - agree the field schema per document type with the client, then build validated extraction that returns nothing rather than a guess where a field is absent.

5. Retrieval pipeline - build the five-step retrieval flow - filter generation, permission enforcement, dual-path search, link expansion, and cited answer composition.

6. The agent layer - wrap retrieval and the other tools in an agent with a capped clarification loop and a capped reasoning loop, so unresolved questions return an honest partial answer.

RESULTS AND IMPACT

CS-015_Document_Intelligence_Toolkit image 4

Figure - Key outcomes from this engagement.

As a proposal-stage engagement, no production results exist yet. The design instead commits to what will be measured before the system reaches real users: permission integrity as a strict pass/fail gate, plus linking coverage, linking accuracy, extraction quality, and answer quality, each scored against examples labeled from the client's own data.

The architecture's central commitments - indexing rather than copying, extracting against an agreed schema rather than a generic hundred-field extractor, and enforcing permissions inside every tool rather than inside the agent - are documented specifically so the system's behavior is auditable and predictable before a single line of production code is written.

What it enabled commercially

If built as proposed, the system would let the client's teams answer a document-grounded question in one search - across whichever system the answer actually lives in - rather than opening several tools in sequence and guessing which of several similarly-named files is the real one.

WHY PFACTORIAL

This engagement draws on Pfactorial's data and pipeline infrastructure capability: designing search and extraction as two distinct problems solved on their own terms, and being explicit about what a proof of concept can and cannot yet claim.

CS-015_Document_Intelligence_Toolkit image 5

Figure - Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with teams whose critical information is scattered across drives, document stores, and meeting recordings with no shared index. If you are evaluating whether a document intelligence system is worth building properly, we are happy to give you an honest read on scope, cost, and risk before anyone commits to anything. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.