OCR & Document Extraction

Reconstructing True Reading Order in Complex, Multi-Format Documents

A layout-aware OCR system built on Tesseract plus a custom reconstruction layer, restoring column, table, and paragraph structure that conventional OCR reads as a single undifferentiated stream.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client works with scanned documents and digital images spanning legal contracts, mineral and property records, financial statements, and other operational documents where layout carries meaning - a two-column agreement, a multi-field table, a numbered clause list. Conventional OCR reads every page as one vertical stream of text, which interleaves unrelated paragraphs and scrambles row-to-value mappings the moment a layout gets more complex than a single column.

Pfactorial Technologies built and validated a layout-aware OCR system that reconstructs a document's true reading order and structure before its text is finalized. The system is built around Tesseract, an open-source OCR engine, paired with a custom layout-reconstruction layer that rebuilds column, row, and paragraph structure from the positional metadata Tesseract already produces for every recognized token.

Rather than replacing Tesseract's proven character-level recognition or retraining a new model, the engineering effort went into the reconstruction logic that runs on top of it - clustering tokens into column bands and row groups, aligning tabular content to a shared grid, and flagging low-confidence spans for review, all without a GPU dependency.

Why this engagement is representative This engagement demonstrates Pfactorial's approach to a common but under-solved document intelligence problem: keeping a proven, pretrained recognition engine for what it does well, and concentrating engineering effort on the structural reconstruction logic that determines whether extracted text actually means anything.

THE CHALLENGE

Scanned and image-based documents rarely arrive in a single, simple column of text, and conventional OCR was not built to handle the layouts that carry real meaning.

1. Side-by-side paragraphs interleave into nonsense

A two-column page read naively is interpreted as a single stream of lines, which merges unrelated paragraphs together and destroys the meaning of each - the most common source of reading-order errors in conventional OCR.

2. Tables lose their row-to-value mapping

Conventional OCR either merges an entire table row into one undifferentiated line, or, on narrower tables, misreads each column as its own vertical stream read independently - scrambling which value belongs to which row and column header.

3. The base OCR engine has no concept of structure

Tesseract's own layout handling stops at reading top-to-bottom, left-to-right in a single stream; it has no built-in notion of columns, tables, or true reading order to fall back on.

4. Source quality varies across every intake channel

Documents arrive as screenshots, scans, and exported reports, each with different resolution, noise, and contrast characteristics that a pipeline tuned to only one capture method cannot reliably handle.

The real brief Not "extract the text on the page" but "preserve the relationships between rows, columns, and paragraphs a human reader would recognize - without retraining or replacing the underlying OCR engine."

THE SOLUTION

Pfactorial kept Tesseract for what it already does well - character-level recognition, exposing per-word position and confidence metadata - and built a dedicated reconstruction layer on top that uses that metadata to group, order, and re-assemble tokens into their correct structure.

Figure 1 - The two-layer design: Tesseract for recognition, a custom reconstruction layer for column, row, and table structure.

Architectural principles

Recognition and reconstruction, kept separate - Tesseract's pretrained models are used as-is for character-level text prediction; the engineering effort is concentrated entirely on the layout-reconstruction logic that runs on its output at inference time, not on retraining the recognizer.
Positional metadata is the raw material - Tesseract exposes left/top coordinates, line and block numbers, and a per-token confidence score alongside predicted text - metadata most "black box" OCR APIs simply do not return, and which the reconstruction layer depends on.
Profile before reconstructing - every incoming page is first classified into single-column text, tabular/numeric content, or mixed/side-by-side layout, and that classification determines which downstream reconstruction rules apply.
Confidence flags uncertainty rather than hiding it - every token's native prediction confidence is used to flag low-confidence spans for review and to disambiguate ambiguous positional clustering, rather than accepting every prediction at face value.

CAPABILITIES DELIVERED

The system's capabilities span the full path from a scanned page to clean, structured, editable text.

CAPABILITY	WHAT IT DOES
Input Document Profiling	Classifies every page into single-column, tabular/numeric, or mixed layout before reconstruction rules are applied.
Column & Paragraph Reconstruction	Clusters word-level bounding boxes into column bands and orders text top-to-bottom within each band, preventing unrelated paragraphs from interleaving.
Table Structure Recovery	Aligns tokens to a shared row and column grid derived from coordinate clustering, keeping each cell tied to its correct row and header.
Confidence-Aware Assembly	Flags low-confidence spans for review and disambiguates ambiguous positional clustering, such as near a column boundary, using per-token confidence scores.
GPU-Free Inference	Processes each page independently on standard CPU infrastructure, keeping deployment straightforward on-premise or in a standard server environment.
REST API Integration	Exposes the pipeline as a service layer that downstream systems can call directly for document conversion.

Figure 2 - Where layout-aware reconstruction plugs in, and the two reading-order failure modes it corrects for.

Design note Recognition accuracy is tied to source scan clarity, and irregular or free-form layouts are harder to cluster deterministically than consistent column or row bands; both are documented, known behaviors with per-token confidence flagging and tunable clustering tolerances as their respective mitigations, rather than being treated as solved.

ENGINEERING FOR SCALE AND RELIABILITY

Several engineering decisions distinguish a reconstruction layer that holds up on real, messy scanned documents from one that only works on a clean demo page.

Coordinate clustering, not layout-model training

columns are identified by clustering word-level bounding boxes on horizontal position, and rows within each column by vertical position - a deterministic, explainable approach that avoids the GPU dependency and training overhead a deep-learning layout-detection model would add.

Table recovery treats rows and columns as one grid

tabular tokens are aligned to a shared row/column grid derived from coordinate and line-block-number clustering, rather than trusting either a pure row-first or pure column-first read in isolation - directly correcting the two failure modes conventional OCR exhibits on tables.

Tunable tolerances for irregular layouts

coordinate-clustering tolerances are tunable per document type, and spans that don't resolve into consistent bands are flagged rather than force-assigned to a column or row they may not belong to.

No line/table boundary detection required from the base engine

because Tesseract itself does not detect lines separating tables from paragraphs, the reconstruction layer infers those boundaries from coordinate and block-number clustering rather than depending on a capability the base engine lacks.

CPU-only by design

each page is processed independently with no GPU dependency, keeping hardware costs and deployment footprint predictable and making the pipeline straightforward to run on standard server or on-premise hardware.

Alternatives evaluated and rejected on evidence

raw Tesseract, cloud OCR APIs, and deep-learning layout-detection models were each evaluated and rejected - for reading-order loss, per-page cost and external data transfer, and unnecessary GPU/training overhead respectively - before the adopted coordinate-based reconstruction approach was selected.

DELIVERY APPROACH

The engagement moved from evaluating OCR and layout-detection approaches to a validated, deployed reconstruction pipeline.

1. Evaluate OCR and layout approaches - compared raw Tesseract, cloud OCR APIs, and deep-learning layout-detection models against layout-preservation requirements before selecting a coordinate-based reconstruction architecture.

2. Build the metadata extraction layer - ran Tesseract in data-output mode to capture per-token text, position, block number, and confidence score for every recognized word.

3. Implement column and paragraph reconstruction - clustered tokens into column bands by horizontal position, then into rows within each band by vertical position, to preserve each paragraph's internal sequence.

4. Implement table structure recovery - aligned tabular tokens to a shared row/column grid derived from coordinate and block-number clustering, rather than a single-axis read.

5. Add confidence-aware assembly - used per-token confidence scores to flag low-confidence spans for review and to disambiguate ambiguous clustering near layout boundaries.

6. Deploy as a GPU-free service - packaged the pipeline behind a REST API on standard CPU infrastructure, validated against real legal, property, and financial source documents.

RESULTS AND IMPACT

Figure - Key outcomes from this engagement.

Against real source documents - a structured property/interest report, a multi-column claims record, and a side-by-side numbered contract clause layout - the reconstructed output preserved field-to-value context, row groupings, and clause integrity that an ordinary, layout-unaware OCR engine lost by merging or scrambling the original structure.

Because the pipeline runs on standard CPU infrastructure with no GPU dependency, it is deployable on-premise or in a standard server environment - a relevant property for clients handling sensitive or regulated documents that cannot be routed through external cloud OCR services.

What it enabled commercially

By preserving layout structure rather than just extracting raw text, the client's teams can convert scanned legal contracts, property records, and financial statements into structured, editable text without a bespoke reconstruction effort per document type, and without depending on external cloud OCR services for sensitive material.

WHY PFACTORIAL

This engagement draws on Pfactorial's data and pipeline infrastructure capability: keeping a proven, pretrained recognition engine for what it already does well, and concentrating engineering effort on the structural reconstruction logic that determines whether extracted text actually preserves meaning.

Figure - Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with teams converting scanned or image-based document archives where layout carries meaning that cannot be lost in digitization. If you are evaluating whether an OCR pipeline needs true layout awareness, we are happy to give you an honest read on scope, cost, and risk before anyone commits to anything. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.

Result and Analysis

ENGAGEMENT SNAPSHOT

A layout-aware OCR system built on Tesseract plus a custom reconstruction layer, restoring column, table, and paragraph structure that conventional OCR reads as a single undifferentiated stream.