ML Infra, Classifiers & RL

Applying Reinforcement Learning and RLHF to Real-World Quality Control

Three delivered systems - a grammar-correction engine trained with RLHF, an RL-governed brand-scoring platform, and a self-healing search pipeline - showing how the same discipline of feedback-driven learning and automated validation applies across different domains.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

CS-031_Reinforcement_Learning_Systems image 1

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client was evaluating how reinforcement learning, reinforcement learning from human feedback, and automated quality control could be applied to validate and continuously improve an existing RL-environment pipeline. Rather than a single new build, the engagement drew on three previously delivered systems - spanning EdTech, e-commerce intelligence, and enterprise search - as concrete evidence for how the same discipline holds up across genuinely different domains.

The most detailed of the three is a Grammar Error Correction system for Australian English, built for an EdTech platform serving school students and English-language learners. A T5-Large model was first fine-tuned on labeled correction examples, then further aligned using Reinforcement Learning from Human Feedback: a reward model trained on English teachers' pairwise rankings, followed by PPO fine-tuning against that reward model with a KL-divergence penalty to prevent the policy from drifting into reward hacking.

The other two systems apply the same underlying discipline differently: an RL agent that continuously re-weights the signals behind brand scoring for a platform ranking over 215,000 Shopify brands, validated by an automated QA framework before any score is published; and a self-healing RLHF loop that uses live user feedback and zero-result search logs to correct a natural-language-to-Elasticsearch query translation layer without manual retraining.

Why this engagement is representative This engagement demonstrates Pfactorial's applied reinforcement learning capability across genuinely different domains: reward modeling and PPO alignment for language generation, RL-governed scoring validated by automated QA at scale, and a self-healing feedback loop for search - each grounded in measured evaluation rather than a single, narrow demo.

THE CHALLENGE

Across all three systems, the underlying problem was the same shape: static rules or a single model's output age poorly as real-world signal, feedback, and edge cases accumulate. Four specific problems shaped how each system was built.

1. Grammar correction has to preserve regional spelling, not just fix errors

A correction system for Australian English cannot simply apply general grammar fixes - it has to actively preserve spellings like "colour," "organise," and "travelled" rather than drifting toward American English defaults, which a generic fine-tuned model does not do reliably.

2. Fixed scoring rules age poorly as markets shift

A brand-intelligence platform that ranks brands on a static rule set over-rates signals like social buzz that don't hold up, and under-corrects as the market changes, eroding trust in the rankings over time.

3. Search quality degrades silently without a feedback signal

A natural-language-to-Elasticsearch translation layer accumulates parsing errors and zero-result queries over time with no mechanism to detect or correct them without manual intervention.

4. Feedback needs to be structured before it can train anything

Raw accept/reject signals, thumbs-down ratings, and zero-result logs are not themselves training data; each system needed a defined process to turn raw feedback into a reward signal or a correction before it could improve anything.

The real brief Not "apply reinforcement learning somewhere" but "use feedback-driven learning and automated validation to keep a system accurate as real-world conditions change, with the reasoning traceable and auditable."

THE SOLUTION

The most fully specified of the three systems - Grammar Error Correction - follows a two-stage training pipeline: supervised fine-tuning to establish a baseline policy, then reinforcement learning from human feedback to align that policy with what human reviewers actually prefer.

CS-031_Reinforcement_Learning_Systems image 2

Figure 1 - The RLHF training pipeline: supervised fine-tuning, reward model training on teacher preferences, and PPO alignment against that reward model.

Architectural principles

Supervised fine-tuning establishes the baseline policy - a pretrained T5-Large checkpoint is fine-tuned as a sequence-to-sequence model on thousands of incorrect-to-corrected sentence pairs, producing the baseline policy (π_SFT) before any reinforcement learning begins.
The reward model learns from pairwise human preference, not a single score - a smaller transformer encoder is trained on English teachers' pairwise rankings of candidate corrections using a Bradley-Terry ranking loss, producing a scalar reward for any input/output pair rather than a single absolute quality label.
PPO aligns the policy without letting it drift - Proximal Policy Optimisation fine-tunes the SFT policy against the reward model's score, with a KL-divergence penalty against the original SFT policy subtracted from the reward specifically to prevent reward hacking.
The same discipline, applied to scoring and to search - the brand-intelligence platform's RL agent and the search platform's self-healing cache both apply the same underlying pattern - observe a signal, act, and get scored against a ground-truth or human-feedback outcome - adapted to a scoring problem and a retrieval problem respectively.

CAPABILITIES DELIVERED

Across the three systems, the capabilities delivered span feedback-driven language correction, RL-governed quality scoring at scale, and self-healing search retrieval.

CAPABILITY	WHAT IT DOES
RLHF-Aligned Grammar Correction	Corrects grammar, spelling, punctuation, and word usage while preserving Australian English spelling, aligned to human teacher preference via PPO.
Per-Category Error Evaluation	Uses ERRANT-style edit extraction to tag and evaluate corrections by category - spelling, tense, agreement, articles, punctuation.
Continuous Reward-Model Refresh	Logs student accept/reject signals and periodically uses them to refresh the reward model, keeping the correction model aligned over time.
RL-Governed Brand Scoring	An RL agent adjusts how live signals - sales velocity, traffic, engagement, ad spend, review sentiment - are weighted per brand, scored against real subsequent outcomes.
Automated Quality Assurance at Scale	Validates every brand score, ranking, and recommendation continuously across 215,000+ brands before publication, rather than spot-checking samples.
Self-Healing Search Cache	Purges negatively rated or persistently zero-result cached queries, forcing a fresh LLM parsing attempt and using top-rated pairs as dynamic few-shot examples.

CS-031_Reinforcement_Learning_Systems image 3

Figure 2 - The same discipline, three domains: feedback-driven correction, RL-governed scoring, and self-healing search.

Design note The three systems are presented together because they demonstrate the same underlying RL/RLHF/QC discipline across different problem shapes - language generation, scoring, and retrieval - not because they share a codebase or a single deployment. Each is a separately delivered system in its own domain.

ENGINEERING FOR SCALE AND RELIABILITY

Several engineering decisions, consistent across the three systems, distinguish feedback-driven learning that holds up in production from a one-off training run.

A separate reward model, not a single fine-tuning pass

the grammar-correction system trains a dedicated reward model on human pairwise comparisons rather than fine-tuning directly on preference labels, giving PPO a stable, reusable scoring function to optimize against.

A KL penalty keeps alignment from degenerating

PPO's objective explicitly subtracts a KL-divergence penalty against the original SFT policy, a standard mitigation against reward hacking where a model over-optimizes against the reward model at the expense of real quality.

RL-governed scoring is validated, not trusted blindly

the brand-intelligence platform's RL agent's rating decisions are logged and validated through the same automated QA framework applied to every other score, keeping the RL agent's reasoning traceable and auditable rather than treated as a black box.

Negative signals actively purge stale cache entries

the search platform's self-healing mechanism doesn't just log negative feedback - it purges the corresponding cache entry, forcing the next matching request back through the LLM for a fresh parsing attempt rather than continuing to serve a known-bad cached result.

Evaluation uses standard, per-category metrics, not a single number

the grammar-correction system is evaluated with GLEU and F0.5 (CoNLL/BEA-style scoring) alongside an Australian-English-specific spelling-retention check and a direct teacher preference study, rather than a single headline accuracy figure.

Inference kept production-ready under load

the grammar-correction model is served behind a containerized FastAPI endpoint with batched, mixed-precision (FP16) inference, and the service was performance-tested under simulated concurrent load using Locust.

DELIVERY APPROACH

Each of the three systems moved through data preparation, model or agent training, and validation before deployment.

1. Prepare and pre-process training data - annotated Australian school essays and a purpose-built golden dataset were normalized, PII-scrubbed, tokenized with SentencePiece, and error-tagged with ERRANT-style extraction.

2. Fine-tune the baseline policy - fine-tuned T5-Large end to end as a sequence-to-sequence correction model using standard cross-entropy loss with teacher forcing.

3. Train the reward model - collected teacher pairwise rankings of SFT-generated candidates and trained a T5-base reward model with a Bradley-Terry pairwise ranking loss.

4. Align the policy with PPO - fine-tuned the SFT policy against the reward model using PPO's clipped surrogate objective with a KL-divergence penalty, producing the deployed π_RLHF policy.

5. Evaluate against standard and domain-specific metrics - compared the RLHF-aligned model to the SFT-only baseline on F0.5, GLEU, Australian-spelling retention, and a direct teacher preference study.

6. Deploy and close the feedback loop - served the model behind a containerized FastAPI endpoint with batched FP16 inference, and logged accept/reject signals to periodically refresh the reward model.

RESULTS AND IMPACT

CS-031_Reinforcement_Learning_Systems image 4

Figure - Key outcomes from this engagement.

On the grammar-correction system, RLHF alignment improved F0.5 correction quality from 0.61 to 0.69 and GLEU from 0.71 to 0.77 against the SFT-only baseline, lifted Australian-spelling retention from 88% to 94%, and won a 73% teacher preference rate against the baseline in direct comparison - with worked examples showing the aligned model correctly enforcing Australian spelling and fixing tense errors the baseline model missed.

On the brand-intelligence platform, continuous RL-governed scoring combined with automated QA validation across more than 215,000 Shopify brands eliminated manual data collection and review, delivering a reported 90% time savings while keeping every rating decision traceable to the ground-truth outcomes that reinforced or penalized it.

What it enabled commercially

Across all three systems, feedback-driven learning replaced static rules or a single fixed model with something that improves as real usage accumulates - reducing the ongoing cost of manual review and re-tuning, while keeping every automated decision logged and auditable rather than opaque.

WHY PFACTORIAL

This engagement draws on Pfactorial's applied research capability: reinforcement learning and RLHF systems engineering that spans reward modeling, policy alignment, and automated quality governance, validated with standard metrics and direct human preference studies rather than a single unverified benchmark.

CS-031_Reinforcement_Learning_Systems image 5

Figure - Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with teams evaluating whether reinforcement learning or RLHF is the right tool for a quality, scoring, or correction problem. If you are weighing feedback-driven learning against simpler static rules, we are happy to give you an honest read on scope, cost, and risk before anyone commits to anything. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.