Healthcare & Clinical

A HIPAA-Compliant De-Identification Pipeline for Multi-Format Clinical Data

How Pfactorial Technologies built a de-identification pipeline that safely anonymizes both raw structured/unstructured clinical files and standardized OMOP CDM records for research use.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

Pfactorial_Case_Study_Data_DeIdentification_Pipeline image 1

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client needed to make two very differently structured clinical datasets - large raw text files and a standardized OMOP CDM database - safely usable for research without exposing any patient-identifiable information.

De-identification is not a single filter. Patient IDs, timestamps, structured fields and unstructured narrative text each carry re-identification risk in different ways, and a method that works for one category - like masking a structured field - does nothing for a narrative discharge summary that mentions a patient's name in free text.

Pfactorial built a four-stage de-identification pipeline - synthetic ID substitution, date offsetting, structured-field masking, and NLP-driven scrubbing of unstructured narrative text - applied consistently across both dataset structures to produce HIPAA-compliant, research-ready data.

Why this engagement is representative This engagement demonstrates Pfactorial's ability to apply a consistent privacy standard across genuinely different data structures - raw flat files and a standardized common data model - rather than building a one-off solution for a single dataset shape.

THE CHALLENGE

Making two structurally different datasets safely usable for research meant solving distinct identification risks across both.

1. Identifiers appear in many forms, not just one field

Medical Record Numbers and Visit IDs appear across tables, files, and research-related materials - every occurrence needs consistent substitution, not just the primary key field.

2. Exact dates are themselves identifying

Precise timestamps can be cross-referenced against other records to re-identify a patient, even when names and IDs are removed - the temporal information itself carries risk.

3. Structured masking does nothing for narrative text

Patient names, addresses and phone numbers embedded in free-text discharge summaries and clinical notes aren't caught by field-level masking rules built for structured data.

4. Two dataset structures, one privacy standard

A large raw flat-file dataset and a standardized OMOP CDM database have completely different schemas, but both needed to meet the same HIPAA-compliant de-identification bar.

The real brief Not “remove the obvious identifiers” but “apply a consistent, defensible de-identification standard across every form an identifier can take, in every dataset structure the client has.”

THE SOLUTION

Pfactorial built a four-stage pipeline, each stage purpose-built for a specific category of identifying information, applied consistently across both the raw flat-file dataset and the OMOP CDM dataset.

Pfactorial_Case_Study_Data_DeIdentification_Pipeline image 2

Figure 2 - Four purpose-built stages, one consistent de-identification standard.

Architectural principles

Synthetic identifiers everywhere an ID appears - Patient and encounter identifiers are substituted with synthetic counterparts across every table, file and research-related material - not just in a primary lookup field.
Relative time, not absolute time - Random offsets are applied to original timestamps, preserving relative date relationships useful for research while removing the ability to trace data back to specific real-world moments.
Field-level masking for structured PHI - Structured fields containing direct identifiers - hospital names, accession numbers, addresses - are masked or anonymized to prevent re-identification from direct data points.
NLP-driven scrubbing for narrative text - The NLM Scrubber tool processes clinical notes and other unstructured text using NLP techniques to identify and remove PHI elements that field-level rules can't catch.

CAPABILITIES DELIVERED

Each capability addresses a distinct category of re-identification risk across both dataset structures.

CAPABILITY	WHAT IT DOES
Synthetic ID generation	MRNs and visit IDs replaced consistently everywhere they appear.
Date obfuscation	Random offsets applied to timestamps, preserving relative research value.
Structured-field masking	Hospital names, addresses and accession numbers anonymized across all structured tables.
Narrative text scrubbing	NLM Scrubber removes names, addresses, phone numbers and MRNs from clinical notes and reports.
Dual dataset coverage	The same four-stage standard applied to both raw flat files and the OMOP CDM dataset.
Configurable PHI targeting	Scrubbing tool customization to target specific PHI elements for removal as needed.

Pfactorial_Case_Study_Data_DeIdentification_Pipeline image 3

Figure 3 - Two very different dataset structures, one consistent de-identification standard.

Design note The two datasets have almost nothing in common structurally - one is raw flat files up to 144 GB, the other is a standardized common data model - but both needed to hit the exact same HIPAA bar. That consistency, not any single technique, is the actual deliverable.

ENGINEERING FOR SCALE AND RELIABILITY

De-identification only counts if it's applied completely and consistently. Five decisions kept the pipeline defensible.

Every-occurrence ID replacement

Original identifiers are located and replaced wherever they appear, not just in a primary key column, closing the gap where a raw ID might otherwise leak through a secondary reference.

Offset-based rather than deleted timestamps

Dates are offset rather than removed entirely, preserving the sequencing and duration information research actually needs while eliminating traceability to real calendar dates.

Purpose-built scrubbing for narrative text

NLM Scrubber, a tool developed specifically for clinical narrative de-identification, is used rather than a generic text-redaction approach, since clinical notes have their own PHI patterns and phrasing.

NLP-driven, not purely rule-based, PHI detection

The scrubbing process leverages natural language processing to accurately parse and anonymize clinical narratives, catching PHI mentioned in varied phrasing that a fixed pattern list would miss.

Consistency across dataset structures

The same four-stage process - ID substitution, date offset, structured masking, narrative scrubbing - is applied to both the raw flat-file dataset and the OMOP CDM dataset, avoiding a weaker standard for either.

DELIVERY APPROACH

The engagement built the pipeline stage by stage, validating each identifier category before moving to the next.

1. Identifier inventory - cataloging every form patient and encounter identifiers take across both dataset structures.

2. Synthetic ID substitution - generating and applying synthetic identifiers consistently across tables, files and research materials.

3. Date obfuscation - applying random offsets to preserve relative temporal information while removing absolute traceability.

4. Structured-field masking - identifying and anonymizing direct identifiers in structured data fields.

5. Narrative text scrubbing - deploying NLM Scrubber against clinical notes and unstructured reports.

RESULTS AND IMPACT

Pfactorial_Case_Study_Data_DeIdentification_Pipeline image 4

Figure 4 - Key outcomes from this engagement.

Both the raw flat-file dataset (~25,000 patient records) and the OMOP CDM dataset (~60,000 patients) are now HIPAA-compliant and usable for research analysis, with no re-identifiable PHI remaining.

A consistent de-identification standard across both dataset structures means the client's research teams can work across either dataset with the same privacy guarantees.

What it enabled commercially

The client can now use both datasets for research and analytics with defensible HIPAA compliance, unlocking the analytical value of over 85,000 patient records that would otherwise be too sensitive to work with directly.

WHY PFACTORIAL

This engagement reflects Pfactorial's health-data engineering service line: building privacy-preserving pipelines that hold up to regulatory scrutiny across genuinely different data structures, not a one-off script for a single dataset.

Pfactorial_Case_Study_Data_DeIdentification_Pipeline image 5

Figure 5 - Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with life-sciences and healthcare organizations that need clinical data made safely usable for research. If you're evaluating a data de-identification or privacy-engineering initiative, we're happy to give you an honest read on scope and risk before anyone commits to anything. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.

Pfactorial_Case_Study_Data_DeIdentification_Pipeline image 6

© 2026 Pfactorial Technologies. Client identity and product-specific implementation detail are withheld or generalized; no client data, credentials, source code, or infrastructure detail is included in this document.

Result and Analysis

ENGAGEMENT SNAPSHOT

How Pfactorial Technologies built a de-identification pipeline that safely anonymizes both raw structured/unstructured clinical files and standardized OMOP CDM records for research use.