OCR & Document Extraction

A Retrieval-Augmented Chat Interface Over Structured Tax Records

How Pfactorial Technologies built a RAG platform that blends database-style lookups with document-grounded search to answer natural-language questions over scanned bills and notices.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

Pfactorial_Case_Study_AI_Search_Engine_Tax_Records image 1

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client manages a large volume of scanned tax bills and notices and wanted staff and users to ask natural-language questions against that corpus - “what's owed on account X” - instead of manually searching records.

The obstacle was that a question like that has two different answers hiding inside it: a database-style fact (an amount, a due date) and unstructured document context (why, from which notice). Most retrieval systems are built for one or the other, not both at once, and OCR-extracted data is rarely clean enough to query directly.

Pfactorial built a proof-of-concept platform that preprocesses OCR output for the specific fields queries actually use, runs a dual retrieval pipeline - a structured Pandas query engine alongside a document context extractor - and synthesizes both into one coherent, conversational answer with memory across turns.

Why this engagement is representative This engagement shows a pattern that shows up constantly in document-heavy industries: the real difficulty isn't running OCR or standing up a chatbot, it's knowing when a question needs a database query versus a document search - and answering both kinds through one interface.

THE CHALLENGE

The client's goal - ask questions, get answers - was simple to state and required solving four distinct problems before it could work reliably.

1. OCR output is not query-ready

Text and tables extracted from scanned bills and notices arrive with shifted columns and stray text. Querying that data directly returns noise, so targeted preprocessing was required before any retrieval could be trusted.

2. Some answers live in a row, others live in a paragraph

A question about a specific tax amount is a database lookup; a question about why a charge appeared is a document-search problem. A single retrieval method can't answer both types of question well.

3. The corpus never stops growing

New bills and notices needed to be searchable immediately after upload, not after an overnight batch job - the client's users couldn't wait for a nightly reindex.

4. Multi-turn conversations need memory

A follow-up question referencing “that account” only works if the system carries forward the context of what was just discussed, not just the literal text of the new query.

The real brief Not “build a chatbot over PDFs” but “build a retrieval system that knows when to run a database-style query versus when to search document text - and blends both into one answer.”

THE SOLUTION

Pfactorial built the platform around targeted preprocessing feeding a dual-retrieval query processor, so structured and unstructured questions are answered through the same conversational interface.

Pfactorial_Case_Study_AI_Search_Engine_Tax_Records image 2

Figure 2 - From scanned document to synthesized, conversational answer.

Architectural principles

Preprocess for the fields that matter - Rather than a generic OCR cleanup pass, preprocessing specifically targets the fields queries actually use - ID, account number, tax year, county, address - repairing shifted columns and removing stray text around them.
Dual retrieval, one interface - A Pandas query engine (built as an agent via LlamaIndex) handles structured lookups against the relational data; a context data extractor pulls matching document text by ID or account number for everything else.
Synthesize, don't concatenate - A response synthesizer combines the outputs of both retrieval paths into one coherent answer, rather than returning two raw fragments for the user to reconcile themselves.
Session memory as a first-class concern - The context extracted for the current query is added to conversation history, so a follow-up question resolves correctly against what was just discussed.

CAPABILITIES DELIVERED

Every capability was scoped around removing a step a user would otherwise have to do manually - searching a filing cabinet, cross-referencing an account number, or waiting for a batch job.

CAPABILITY	WHAT IT DOES
Custom document upload	Users upload their own PDFs, each tracked with a unique ID extracted directly from the document.
Real-time ingestion	Newly uploaded documents are searchable immediately, with no batch delay.
Conversational Q&A	Natural-language queries against the corpus, with adjustable response-accuracy (model) settings.
Feedback loop	Like/dislike feedback on responses is tracked per query, including changes to that feedback over time.
Read-aloud & copy	Responses can be read aloud or copied directly from the chat interface.
Secured session access	JWT-based authentication with admin-provisioned accounts - no public self-signup.

Pfactorial_Case_Study_AI_Search_Engine_Tax_Records image 3

Figure 3 - Structured and semi-structured retrieval, fused into one response.

Design note Two retrieval components instead of one is the whole design decision. A structured-only system can't explain a charge; a document-search-only system can't total an amount. Neither replaces the other.

ENGINEERING FOR SCALE AND RELIABILITY

The gap between a demo and a usable research tool is in the preprocessing and retrieval details that don't show up in a first pass.

Targeted data preprocessing

Specific, recurring OCR errors - reordered columns, injected stray text - are corrected for the fields the query engine depends on, rather than attempting a generic cleanup of the entire extraction.

Agentic structured retrieval

The Pandas query engine acts as an agent that generates the relevant query for a given question and synthesizes a response from structured data, rather than requiring users to know the schema.

Two-tier document storage

Structured fields go to a relational database while combined text and table data is written per-document to the filesystem, with a reference retained for fast lookup.

Trackable feedback signal

Feedback defaults to neutral and updates are tracked as a change history, not just a final state - so shifts in user sentiment on a response are visible, not overwritten.

Controlled account provisioning

The application intentionally has no self-service sign-up; accounts are added by administrators, keeping access to a sensitive tax-records corpus deliberately gated.

Containerized, portable deployment

Frontend and backend ship as separate Docker images, saved as portable tar files rather than requiring a live registry for distribution.

DELIVERY APPROACH

The proof of concept was sequenced to prove the hardest part - dual retrieval accuracy - before layering on the surrounding product experience.

1. Data pipeline assessment - evaluation of OCR output quality and identification of the specific preprocessing needed for reliable structured-field extraction.

2. Query processor architecture - the Pandas query engine, context data extractor and response synthesizer, built and validated against the sample question set.

3. Chat interface & auth - the React front end, JWT-secured session flow, and PDF upload/preview experience.

4. Feedback instrumentation - like/dislike tracking with change history, to give the client visibility into response quality over time.

5. Containerization - Dockerized frontend and backend images for portable deployment and handover.

RESULTS AND IMPACT

Pfactorial_Case_Study_AI_Search_Engine_Tax_Records image 4

Figure 4 - Key outcomes from this engagement.

The client's users can ask natural-language questions across a corpus of tax bills and notices and receive answers that blend structured facts with the document context behind them, instead of manually cross-referencing records.

The feedback loop gives the client an ongoing, queryable signal on answer quality as the system is used, rather than relying on anecdotal reports.

What it enabled commercially

The proof of concept demonstrated that a single conversational interface could answer both the database-style and document-grounded questions the client's users actually ask, validating the approach for further investment beyond the POC stage.

WHY PFACTORIAL

This engagement reflects Pfactorial's applied RAG engineering service line: not a generic chatbot-over-documents integration, but a retrieval architecture designed around the specific shape of the client's data and the specific questions their users ask of it.

Pfactorial_Case_Study_AI_Search_Engine_Tax_Records image 5

Figure 5 - Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with organizations sitting on large volumes of structured and semi-structured documents who want those documents queryable in natural language. If you're evaluating what a retrieval-augmented interface would take to build over your own records, we're happy to give you an honest read on scope and risk before anyone commits to anything. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.

Pfactorial_Case_Study_AI_Search_Engine_Tax_Records image 6

© 2026 Pfactorial Technologies. Client identity and product-specific implementation detail are withheld or generalized; no client data, credentials, source code, or infrastructure detail is included in this document.

Result and Analysis

ENGAGEMENT SNAPSHOT

How Pfactorial Technologies built a RAG platform that blends database-style lookups with document-grounded search to answer natural-language questions over scanned bills and notices.