OCR & Document Extraction

Reading Handwritten Dutch Text at Production Accuracy

A fine-tuned transformer OCR pipeline, purpose-built for Dutch handwriting, that outperforms general-purpose multimodal tools and improves continuously through a built-in correction feedback loop.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

CS-021_Handwritten_Dutch_OCR_System image 1

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client needed to digitize handwritten Dutch documents - historical archives, legal and medical notes, business correspondence - at a level of accuracy general-purpose OCR tools, including multimodal models like GPT-4's vision capabilities, could not reliably deliver on Dutch-language handwriting.

Pfactorial Technologies built and deployed a specialized OCR pipeline that fine-tunes a transformer OCR model specifically for Dutch handwriting, wraps it in a FastAPI service layer with built-in preprocessing and error handling, and captures every user correction into a feedback store that feeds ongoing model refinement.

The system was validated against a held-out benchmark spanning historical manuscripts, legal notes, and business correspondence, exceeding 85% accuracy against a Tesseract baseline while processing each image in under two seconds - with a continuous feedback loop designed so accuracy compounds over time rather than staying fixed at its initial training baseline.

Why this engagement is representative This engagement demonstrates Pfactorial's approach to specialized OCR: fine-tuning a purpose-built model against a curated, language-specific dataset rather than relying on a general-purpose multimodal tool, and validating the result against a real, held-out benchmark rather than a single hand-picked example.

THE CHALLENGE

General-purpose OCR tools were not built for the specific difficulty of Dutch handwriting, and closing that gap meant solving several problems together.

1. General-purpose OCR falls short on handwriting

General-purpose multimodal tools, including GPT-4's vision capabilities, and traditional engines like Tesseract, fell short of production accuracy requirements on real Dutch handwritten text, confirming the need for a fine-tuned, language-specific model.

2. Handwriting style varies enormously across documents

Historical manuscripts, legal notes, and business correspondence each carry different handwriting styles, document ages, and paper conditions, requiring a training dataset broad enough to cover that real variation.

3. Contemporary Dutch handwriting was underrepresented

Public handwriting corpora like IAM, Nederlab, and READ-BAD gave broad style and historical coverage, but needed to be supplemented with custom-annotated samples to properly represent contemporary Dutch handwriting.

4. Accuracy needed to improve after deployment, not just at launch

A model's accuracy on a specific client's real document population is different from its accuracy on a benchmark set, and needed a mechanism to keep improving as real usage continued, rather than staying fixed at its initial training baseline.

The real brief Not "run OCR on a scanned page" but "read real, messy Dutch handwriting at a production-usable accuracy, and keep getting better at it as real documents come in."

THE SOLUTION

Pfactorial built the system as a five-stage processing flow: preprocessing and segmentation, fine-tuned model inference, extracted text delivery, and a feedback loop that captures corrections for ongoing refinement.

CS-021_Handwritten_Dutch_OCR_System image 2

Figure 1 - End-to-end processing flow: upload, preprocessing, OCR inference, extracted text, optional correction, and feedback storage.

Architectural principles

Fine-tune a purpose-built model, don't rely on general-purpose vision - a TrOCR checkpoint was fine-tuned specifically on Dutch handwriting with LoRA, chosen over general multimodal vision models because its encoder-decoder design is purpose-built for line-level handwriting recognition rather than general document understanding.
Parameter-efficient fine-tuning keeps the compute budget controlled - LoRA adaptation was applied to keep the compute footprint low while adapting the pretrained model to Dutch handwriting, rather than fine-tuning the full model end to end.
Preprocessing does real work before inference runs - deskewing, noise removal, and CLAHE contrast enhancement normalize scan orientation and stabilize recognition across faded or unevenly lit source material before the transformer model ever sees the image.
Every correction becomes training signal - corrections submitted through the API are structured and persisted for direct use in retraining, with an active learning pipeline prioritizing misclassified and corrected samples during each refinement cycle.

CAPABILITIES DELIVERED

The delivered system covers the full path from an uploaded image to structured, correctable text output.

CAPABILITY	WHAT IT DOES
Handwritten Text Extraction	Uploads a handwritten Dutch document image and returns extracted text as structured JSON via a web-based API.
Fine-Tuned Transformer OCR	A TrOCR model fine-tuned with LoRA on curated Dutch handwriting corpora, exceeding traditional engines like Tesseract on this task.
Production Service Layer	A FastAPI service with built-in image preprocessing, structured error handling, and request logging for every call.
Confidence-Scored Output	Every OCR response carries a confidence score, so low-certainty results can be flagged for human review rather than accepted silently.
Correction Feedback Capture	User-submitted corrections are stored in a dedicated feedback store, structured for direct use in scheduled retraining cycles.
Optimized Inference Deployment	Served through NVIDIA Triton Server in a Docker container for optimized inference throughput, with a Streamlit demo interface available separately.

CS-021_Handwritten_Dutch_OCR_System image 3

Figure 2 - Deployment surface: real-world use cases the pipeline is applied to, and how the feedback loop keeps accuracy improving.

Design note The delivered model is fine-tuned specifically for Dutch; support for additional languages (French, German, and others) reuses the same fine-tuning pipeline applied to a new language-specific dataset, offered as a scoped extension rather than a change to the core architecture.

ENGINEERING FOR SCALE AND RELIABILITY

Several engineering decisions distinguish a model that generalizes to real client documents from one that only performs well on a curated benchmark.

TrOCR was selected on architectural fit, not familiarity

TrOCR, LayoutLMv3, and BLIP-2 were each evaluated; LayoutLMv3's layout-understanding overhead and BLIP-2's captioning-oriented training objective were both found unnecessary or poorly matched for pure handwriting transcription.

Line segmentation runs ahead of transformer inference

Tesseract's layout engine handles initial line segmentation before the fine-tuned transformer model runs, splitting the segmentation and recognition jobs between the tool best suited to each.

The feedback loop is structural, not incidental

corrections are stored in a dedicated feedback database (MongoDB / PostgreSQL) and structured specifically for retraining, with an active learning pipeline prioritizing the samples that were actually misclassified or corrected.

Confidence scoring gates trust in every response

the API surfaces a confidence score with every response specifically so downstream systems can flag low-confidence results for human review rather than accepting a low-quality transcription silently.

Known constraints are documented, not discovered later

handwriting style variance, language scope, structured-layout handling, and low-quality scan confidence are each documented as known constraints with a specific, already-implemented mitigation.

Deployment is optimized for sustained throughput

the model is served through NVIDIA Triton Server specifically for optimized inference throughput under deployment load, validated at 99.9% API uptime rather than assumed.

DELIVERY APPROACH

The engagement moved from dataset curation to a validated, deployed OCR service in five phases.

1. Curate the training dataset - assembled training data from the IAM, Nederlab, and READ-BAD corpora, supplemented with custom-annotated contemporary Dutch handwriting samples via LabelImg.

2. Build the preprocessing pipeline - implemented deskewing, noise removal, CLAHE contrast enhancement, and initial line segmentation ahead of transformer inference.

3. Fine-tune the core model - fine-tuned a pretrained TrOCR checkpoint on the curated Dutch dataset using LoRA, benchmarked against LayoutLMv3 and BLIP-2 before selection.

4. Build the service and feedback layers - wrapped the model in a FastAPI service with logging and error handling, and built the correction feedback store and active learning pipeline.

5. Validate and deploy - validated against a held-out benchmark spanning historical, legal, and business documents, then deployed via Docker and NVIDIA Triton Server.

RESULTS AND IMPACT

CS-021_Handwritten_Dutch_OCR_System image 4

Figure - Key outcomes from this engagement.

Validation against the held-out benchmark set - spanning historical manuscripts, legal notes, and business correspondence - showed OCR accuracy exceeding 85% on Dutch handwritten text, with processing latency under two seconds per image and a consistent improvement over the Tesseract baseline across every tested document category.

The deployed API sustained 99.9% uptime under deployment load, and the continuous feedback loop was validated as a working mechanism: corrected samples are captured and structured for retraining, so accuracy on a client's specific document population is designed to improve with continued usage.

What it enabled commercially

By exceeding 85% accuracy on real Dutch handwriting - well beyond what general OCR tools deliver - the client's libraries, legal, and medical back-office teams can digitize handwritten archives and intake forms directly through an API, without a person manually transcribing documents one at a time.

WHY PFACTORIAL

This engagement draws on Pfactorial's data and pipeline infrastructure capability: fine-tuning purpose-built models for a specific language and document domain, validated against a real held-out benchmark, with a feedback loop designed to keep improving in production rather than staying fixed at launch.

CS-021_Handwritten_Dutch_OCR_System image 5

Figure - Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with libraries, legal, and medical teams digitizing handwritten archives and forms. If you are evaluating whether a specialized OCR system is worth building properly, we are happy to give you an honest read on scope, cost, and risk before anyone commits to anything. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.

CS-021_Handwritten_Dutch_OCR_System image 6

Result and Analysis

ENGAGEMENT SNAPSHOT

A fine-tuned transformer OCR pipeline, purpose-built for Dutch handwriting, that outperforms general-purpose multimodal tools and improves continuously through a built-in correction feedback loop.