Medical Data into Actionable Insights

Unlocking the Power of Clinical & Genomic Data for Advanced Research and Precision Medicine

May 18, 2026

OUTLINE

CLIENT

Leading medical research institution (USA)

INDUSTRY

Healthcare & Precision Medicine

DATA SCALE

100GB+ EHR | 50K+ OMOP Patients | 2K+ Omics Samples

Executive Summary

Modern healthcare research generates data at a scale and complexity that overwhelms traditional data management approaches. Electronic Health Records (EHRs), genomic sequencing outputs, clinical imaging, and unstructured clinical notes all represent extraordinary research value but only when they can be reliably ingested, standardized, integrated, and analyzed in a unified, compliant framework.

A leading research institution in the United States faced exactly this challenge. Managing over 100GB of EHR data across 20,000+ patient records, standardized OMOP CDM data for 50,000+ patients, and DNA/RNA sequencing data from 2,000+ patients sourced from major US hospitals, their data ecosystem had grown beyond what existing tools and manual processes could handle.

Pfactorial was engaged to design and deliver an end-to-end data analytics platform purpose-built for clinical and genomic research. The solution spans data ingestion and preprocessing, PHI de-identification, ETL standardization, multi-omics processing, advanced ML/AI analytics, and researcher-facing tools that make complex data accessible to non-technical clinicians and scientists.

The result is a scalable, HIPAA-aligned, cloud-hosted research data platform that accelerates biomarker discovery, enables clinical trial cohort identification, and supports precision medicine applications across cardiology, oncology, and pulmonology.

Key Objectives

The engagement was scoped around eight strategic objectives, each addressing a distinct research and operational need:

Efficient Medical Data Processing

Automate and streamline the ingestion, preprocessing, and transformation of large-scale clinical and genomic datasets to reduce manual effort and processing time.

Seamless Data Integration & Standardization

Unify diverse data sources under common medical ontologies (ICD, LOINC, SNOMED) to ensure interoperability across clinical and research systems.

Ensure Data Security & Compliance

Implement HIPAA-compliant PHI de-identification, role-based access control, encryption, and audit logging to protect sensitive patient data.

Accelerate Clinical Research & Cohort Discovery

Enable rapid identification of patient cohorts for clinical trials and observational studies through i2b2 and AI-powered query tools.

ML/AI-Driven Advanced Analytics

Deploy machine learning, deep learning, and statistical models to extract predictive insights from clinical and omics data.

Facilitate Non-Technical Research Access

Develop natural-language and AI-powered interfaces (Text-to-SQL, RAG) so non-technical clinicians and researchers can access complex data without coding.

Optimize Performance & Efficiency

Engineer high-throughput pipelines capable of processing 100GB+ datasets with parallel computing and optimized database architecture.

Scalable & Future-Proof Data Ecosystem

Build a modular, cloud-hosted infrastructure on Google Cloud that scales with growing data volumes and accommodates new research domains.

Data Overview: Volume, Variety & Velocity

The platform operates on a large, heterogeneous, and continuously growing dataset sourced from clinical institutions, genomic sequencing laboratories, and medical knowledge repositories. Understanding the three dimensions of this data challenge is essential context for the solution architecture.

Volume — The Scale of the Data

• EHR Data: Over 100GB of structured electronic health record data encompassing 20,000+ patient records, including demographics, clinical encounters, laboratory results, medication histories, diagnoses, and procedure records.

• OMOP CDM: Standardized clinical data for 50,000+ patients mapped to the Observational Medical Outcomes Partnership (OMOP) Common Data Model — enabling federated research and cross-institutional interoperability.

• Omics Data: DNA and RNA sequencing data from 2,000+ patients, including whole-genome sequencing outputs, RNA expression profiles, and variant call files produced by Illumina sequencing instruments.

Variety — The Diversity of Data Types

• Structured Clinical Data: Demographic fields, ICD-coded diagnoses, LOINC-coded laboratory values, medication records (NDC codes), and procedure data all suited to relational database storage and SQL-based querying.

• Unstructured Clinical Text: Physician narrative reports, discharge summaries, radiology reads, pathology notes, and consult letters requiring NLP-based extraction to derive structured insights.

• Genomic & Omics Data: Raw FASTQ sequencing files, processed VCF variant files, RNA-Seq transcript abundance files, and gene expression matrices processed through specialized bioinformatics pipelines.

• Institutional Data: Multi-site data from major US hospitals, introducing variability in data formats, coding practices, and documentation standards across institutions.

• Medical Reference Resources: ICD-10 code hierarchies, SNOMED CT ontology files, LOINC terminology databases, and curated research articles used for ontology mapping and knowledge enrichment.

Velocity — The Rate of Data Growth

• Incremental Data Growth: New patient records are added continuously as patients move through clinical care. The pipeline is designed for incremental ingestion, processing only new or updated records rather than reprocessing the full dataset on each run.

• Continuous Multi-Source Ingestion: Clinical data arrives from hospital information systems on rolling schedules. Genomic sequencing data is produced in batches from laboratory instruments. The platform handles both streaming and batch ingestion patterns.

Solution Architecture: The End-to-End Data Pipeline

Pfactorial designed a modular, layered solution architecture connecting raw data sources through preprocessing, storage, and analytics to actionable research outputs. The architecture is organized across five functional layers:

Layer 1: Data Sources

Three categories of primary data feed the platform. Clinical data encompasses structured EHR records and OMOP CDM datasets from hospital systems. Omics data includes raw sequencing outputs from Illumina instruments covering both genomic (DNA) and transcriptomic (RNA) profiles. Medical resources include ICD, SNOMED, and LOINC ontology files and curated biomedical literature used for coding standardization and knowledge enrichment.

Layer 2: Preprocessing — De-identification & Extraction

All incoming data passes through a preprocessing layer with two primary functions. De-identification removes or replaces Protected Health Information (PHI) from clinical text using a fine-tuned RoBERTa NER model. Extraction converts unstructured clinical narratives and structured data fields into normalized, analysis-ready formats using NLP pipelines (cTAKES, SpaCy, NLTK) and regex-based parsers.

Layer 3: Technologies Applied

The preprocessing and analytics layers are powered by six technology categories: machine learning for tabular clinical prediction tasks; NLP for clinical concept and entity extraction; deep learning (CNNs, Transformers) for imaging analysis and advanced NLP; computer vision for medical image processing; Retrieval-Augmented Generation (RAG) for natural-language Q&A over clinical knowledge; and Illumina sequencing pipelines for raw genomic data processing.

Layer 4: Data Storage — Hybrid Multi-Database Architecture

Processed data is stored across three complementary database technologies. An SQL Database (MS SQL Server) stores structured relational data including patient records, OMOP CDM tables, and laboratory values. A Graph Database (Neo4j) models complex relationships between medical entities including patient-phenotype-gene-drug networks and ontological concept hierarchies. A Vector Database (Chroma) stores dense embeddings of clinical text and medical concepts, enabling semantic similarity search and RAG-based retrieval.

Layer 5: Analytical Applications & Targeted Actions

The storage layer feeds three primary clinical insight applications: i2b2 for cohort discovery and clinical data warehousing; RStudio for biostatistical and genomic analysis; and LifeOmic for genomics and precision medicine workflow management. These applications produce four categories of targeted research output: precision medicine insights, predictive risk models, drug discovery signals, and medical image analysis results.

PHI De-identification: Outperforming the NLM Benchmark

Clinical text documents — physician notes, discharge summaries, radiology reports — contain rich research-relevant information but also sensitive patient identifiers that must be removed before data can be used for research purposes. HIPAA Safe Harbor and Expert Determination standards define the categories of PHI that must be de-identified, including patient names, dates, geographic identifiers, medical record numbers, and other direct and quasi-identifiers.

Pfactorial developed a custom PHI de-identification model based on fine-tuned RoBERTa — a state-of-the-art transformer architecture fine-tuned on a custom dataset of annotated clinical documents. The model identifies and tags PHI entities using Named Entity Recognition (NER), which are then replaced with synthetic placeholders that preserve the semantic structure of the text for downstream NLP tasks.

Performance was benchmarked against NLM-Scrubber, the PHI de-identification tool developed by the National Library of Medicine. The results demonstrate significant improvements across all evaluation metrics:

Transforming Medical Data into insights Table 1

Our fine-tuned RoBERTa model achieved a Recall of 94.34% (vs. 79.75% for NLM-Scrubber), a Precision of 93.17% (vs. 88.73%), and an F1-Score of 93.75% (vs. 84.00%). The recall improvement is particularly significant in a healthcare compliance context: missed PHI represents a regulatory risk, and a 14.6 percentage point improvement means substantially fewer patient identifiers escape detection. Critically, this was achieved while also improving precision meaning fewer false positives that would otherwise redact useful clinical terms.

Data Extraction: NLP-Powered Clinical Concept Mining

Unstructured Clinical Text Extraction

Clinical narrative documents contain clinical observations not recorded in any structured field. Extracting this information requires natural language understanding tailored to medical terminology and clinical documentation conventions.

The extraction pipeline uses Apache cTAKES (Clinical Text Analysis and Knowledge Extraction System), a medical NLP framework specifically designed for clinical text processing. cTAKES identifies clinical concepts and maps them to SNOMED CT, UMLS CUI codes, RxNorm, and LOINC. For each concept extracted, the system captures the concept code, coding scheme, UMLS CUI identifier, polarity (affirmed or negated), and uncertainty status enabling downstream filtering of negated or uncertain clinical findings.

SpaCy and NLTK complement cTAKES for tokenization, sentence boundary detection, dependency parsing, and custom entity recognition tasks. Together, these tools enable structured data extraction from free-text narratives that previously required manual clinical chart review.

Structured Data Extraction via Regex Pipelines

For data fields with predictable patterns laboratory values, vital signs, medication dosages, measurement units regex-based extraction pipelines provide faster, deterministic extraction with high precision. A comprehensive metadata schema defines allowable value ranges, unit normalization rules, and exception handling for each variable type. For example, mean Pulmonary Artery Pressure (mPAP) values are normalized to mmHg, forced vital capacity (FVC) to percentage, and hemodynamic measurements to their canonical units ensuring comparability across institutions with different documentation conventions.

Omics Data Processing: From Raw Sequences to Biological Insight

Genomic DNA Processing — Illumina Dragon Pipeline

Whole-genome and targeted sequencing data is processed through Illumina's DRAGEN (Dynamic Read Analysis for GENomics) bioinformatics platform. The workflow begins with raw FASTQ files from Illumina sequencing instruments and produces analysis-ready Variant Call Format (VCF) files through three sequential stages.

Alignment maps sequencing reads to the human reference genome (GRCh38), handling read quality filtering, adapter trimming, and duplicate marking. Variant calling identifies SNPs, insertions and deletions (indels), and structural variants relative to the reference genome. Annotation enriches called variants with functional information including population allele frequencies from gnomAD, predicted functional effects from ClinVar and SIFT/PolyPhen, and gene-level annotations linking variants to known disease associations. The resulting annotated VCF files feed downstream ML models for disease risk prediction and genomic association studies.

Transcriptomic RNA Processing — Salmon Pipeline

RNA sequencing (RNA-Seq) data is processed through Salmon, a fast and accurate transcript quantification tool that estimates gene expression levels directly from raw sequencing reads. Salmon outputs transcript-level abundance estimates (TPM and estimated count formats) imported into R via tximport for downstream analysis.

Differential expression analysis is conducted using DESeq2 and edgeR to identify statistically significant differentially expressed genes (DEGs) between patient groups. Gene Set Enrichment Analysis (GSEA) contextualizes these findings within known biological pathways, identifying dysregulated processes including inflammation, vascular remodeling, and right ventricular stress response pathways. These analyses directly support the Gene Expression in PAH study.

Data Integration & Standardization

Clinical research requires data from multiple sources to be unified into a consistent analytical framework. Pfactorial designed and implemented a comprehensive integration and standardization architecture addressing four key challenges:

ETL Pipeline Architecture

Extract-Transform-Load (ETL) pipelines implemented in T-SQL, Python, R, and bash scripting handle the movement of data between source systems and the unified research data platform. Pipelines are designed for incremental processing — only new or changed records are processed on each run — minimizing compute costs and ensuring timely data availability. Error handling, audit logging, and retry logic ensure pipeline reliability in production operation.

Medical Ontology Mapping

All clinical data is mapped to standardized medical terminologies to ensure consistent concept representation across source institutions. Diagnoses are coded to ICD-10-CM. Laboratory observations are mapped to LOINC codes with unit normalization applied to ensure comparable values across institutions. Procedures are mapped to CPT codes. Medications are standardized to RxNorm. SNOMED CT is used for clinical finding and procedure concepts within the OMOP CDM framework. This ontology mapping enables cross-institutional comparisons that would otherwise be confounded by local coding variations.

De-duplication Pipelines

A persistent challenge in multi-site clinical data is duplicate records — the same patient encounter, laboratory result, or diagnosis appearing multiple times due to data export artifacts, system migrations, or patient transfers between institutions. Pfactorial designed specialized de-duplication pipelines tailored to each data type. Laboratory value de-duplication uses timestamps, result values, and LOINC codes to identify and consolidate duplicate measurements. Diagnosis de-duplication applies encounter-level temporal logic to distinguish repeat diagnoses from duplicate records. Each pipeline preserves genuine repeat measurements while eliminating data quality artifacts.

Harmonization & Validation

Harmonization applies consistent transformation rules across all source datasets to produce a unified analytical schema. Validation checks enforce data quality constraints: range checks on numerical values, referential integrity across linked tables, completeness requirements for mandatory fields, and cross-field consistency rules. Validation failures are logged for investigation and trigger data quality alerts to the research team.

Advanced Analytics & Research

The processed and integrated data platform enables a rich suite of analytical capabilities, spanning classical biostatistics, machine learning, deep learning, and AI-powered knowledge synthesis.

Statistical Modeling in RStudio

RStudio serves as the primary environment for biostatistical analysis. Survival analysis using Kaplan-Meier estimators produces time-to-event curves for disease progression, treatment response, and mortality outcomes. Cox Proportional Hazards regression models quantify relationships between clinical and genomic covariates and survival outcomes, adjusting for confounders. Patient stratification techniques segment patient populations into clinically meaningful subgroups for comparative effectiveness research. ggplot2 produces publication-quality visualizations suitable for research paper submission.

Cohort Discovery with i2b2

i2b2 (Informatics for Integrating Biology and the Bedside) provides a structured query interface over the clinical data warehouse, enabling researchers to define and retrieve patient cohorts based on complex combinations of clinical criteria — diagnoses, medications, laboratory values, procedure codes, demographics, and temporal relationships. Cohort discovery in i2b2 directly accelerates clinical trial design by allowing researchers to assess enrollment feasibility against the real patient population before study initiation, reducing trial design iteration cycles significantly.

Machine Learning for Predictive Modeling

Classical machine learning models — Random Forest, XGBoost, and Logistic Regression — are applied to structured EHR and genomic features for disease risk prediction and patient classification. Models are developed using cross-validation to assess generalizability, with feature importance analysis identifying the clinical and genomic variables most predictive of outcomes. Scikit-learn, caret, and tidymodels provide the modeling infrastructure across Python and R environments.

Deep Learning for Imaging & Genomics

Convolutional Neural Networks (CNNs) are deployed for medical imaging analysis tasks, including a mammography classification model distinguishing BIRADS 0 (inconclusive) from BIRADS 1/2 (negative or benign findings). Deep learning models are also applied to genomic feature representations for variant effect prediction and multi-omics integration tasks.

Multi-Omics Integration & Biomarker Discovery

The platform integrates clinical, genomic, and transcriptomic data to support multi-omics research — combining molecular and clinical signals to identify biomarkers with stronger predictive power than either data type alone. This integration supports disease risk assessment, personalized treatment selection, and the identification of novel therapeutic targets from correlated clinical phenotypes and molecular profiles.

Real-Time Research Dashboards

Interactive dashboards built on the integrated data platform provide research teams with real-time visibility into cohort characteristics, data quality metrics, model performance indicators, and analytical outputs. Dashboards support both operational monitoring (pipeline health, data completeness) and research decision-making (cohort size, covariate distributions, preliminary outcome estimates).

Secure Data Management & Compliance

Clinical research data is among the most sensitive categories of personal information. The platform implements a defense-in-depth security architecture addressing four critical control domains:

SSL/TLS Encryption

All data in transit between client applications, APIs, and backend services is encrypted using Transport Layer Security (TLS). SSL certificates enforce HTTPS for all web-based interfaces and API endpoints, protecting data from eavesdropping and man-in-the-middle attacks during transmission. This applies to data flowing between hospital source systems, cloud processing infrastructure, and researcher-facing applications.

Multi-Factor Authentication (MFA)

All platform users are required to authenticate using multiple factors combining a password with a one-time password (OTP) via authenticator app or biometric verification. MFA significantly reduces the risk of unauthorized access from compromised credentials, a leading cause of healthcare data breaches. MFA is enforced platform-wide and cannot be bypassed by individual users.

Role-Based Access Control (RBAC)

System access is governed by a least-privilege RBAC model. User roles correspond to research functions: data engineers have write access to pipeline infrastructure but no access to patient-level data; clinical researchers have read access to de-identified datasets relevant to their approved studies; administrators have audit-logged access to sensitive data. Role assignments are reviewed periodically and revoked upon project completion or personnel changes.

Audit Logs & Access Monitoring

Comprehensive audit logging captures all user authentication events, data access requests, query executions, administrative actions, and system configuration changes. Logs are centrally aggregated and tamper protected. Real-time monitoring alerts the security team to anomalous access patterns unusual query volumes, access outside normal working hours, or attempts to access unauthorized data categories enabling rapid incident response.

Key Studies Conducted

The platform powered five primary research studies, each demonstrating a distinct analytical capability enabled by the integrated data infrastructure:

Study 1: Pulmonary Hypertension (PH) Risk Prediction

Objective: Develop a predictive model for early identification of Pulmonary Hypertension patients from EHR data, reducing the diagnostic delay that characterizes PH — a disease where patients typically receive a diagnosis years after symptom onset.

Approach: The study integrated invasive hemodynamic measurements from Right Heart Catheterization (RHC) — the gold standard diagnostic procedure for PH — with non-invasive clinical features including echocardiography parameters, biomarkers (NT-proBNP, troponin), pulmonary function test results, and medication histories. Machine learning models were trained to predict RHC-confirmed PH from the non-invasive feature set, enabling risk stratification before the invasive procedure.

Study 2: Genomic Risk in Renal Cell Carcinoma (RCC)

Objective: Investigate the contribution of single nucleotide polymorphisms (SNPs) to individual genetic risk of Renal Cell Carcinoma, identifying genomic variants associated with disease susceptibility.

Approach: Whole-genome sequencing data from RCC patients and matched controls was processed through the Illumina Dragon pipeline to produce annotated variant call files. SNPs in candidate genomic regions were analyzed for association with RCC diagnosis using logistic regression and machine learning models, adjusting for population stratification using principal component analysis (PCA). Significant associations were cross-referenced against ClinVar and population allele frequency databases to assess clinical relevance.

Study 3: Pulmonary Arterial Hypertension (PAH) Patient Classification

Objective: Build an early detection framework for Pulmonary Arterial Hypertension — a severe sub-form of PH — to enable treatment initiation before irreversible right heart failure occurs.

Approach: The classification framework integrated medical history variables, treatment response indicators (vasodilator test results, initial therapy outcomes), and hemodynamic parameters. Machine learning models (XGBoost, Random Forest) were trained to classify patients into PAH and non-PAH PH subtypes, with survival analysis assessing the prognostic relevance of classification accuracy. The framework supports clinical decision-making at the point of initial PH diagnosis.

Study 4: Gene Expression Analysis in PAH

Objective: Identify differentially expressed genes (DEGs) in PAH patients compared to healthy controls, providing molecular insights into disease pathophysiology and potential therapeutic targets.

Approach: RNA-Seq data from PAH patient blood samples and controls was processed through the Salmon quantification pipeline and imported into R via tximport. Differential expression analysis using DESeq2 identified statistically significant DEGs. Gene Set Enrichment Analysis (GSEA) contextualized these findings within known biological pathways, identifying dysregulated processes including inflammation, vascular remodeling, and right ventricular stress response pathways.

Study 5: Deep Learning for Mammography Classification

Objective: Develop a computer vision model to assist radiologists in classifying mammography findings — specifically distinguishing BIRADS 0 (inconclusive, requiring additional imaging) from BIRADS 1 and 2 (negative and benign) assessments.

Approach: A Convolutional Neural Network (CNN) was trained on a curated mammography image dataset with radiologist-verified BIRADS labels. Transfer learning from pre-trained imaging models was applied to improve generalization. The model provides decision-support predictions with confidence scores, designed to augment radiologist judgment rather than replace it — flagging cases where additional imaging is likely required.

Empowering Non-Technical Users with AI & Automation

A critical barrier to wider adoption of clinical data platforms is the requirement for SQL or programming expertise to query and analyze complex databases. Pfactorial developed three AI-powered tools that bring the full power of the integrated data platform to clinicians and researchers without technical data skills:

Text-to-SQL AI Tool

Researchers and clinicians can pose questions in plain English — for example, 'How many patients with PAH had an NT-proBNP above 1000 at baseline?' — and receive structured database query results without writing SQL. The tool uses a large language model fine-tuned on the platform's schema to translate natural-language questions into accurate SQL queries, executes them, and returns results in readable format. This democratizes data access, enabling clinical researchers to explore hypotheses independently without requesting analyst support for every data query.

Medical Code Retrieval System

Medical coding — mapping clinical concepts to ICD, SNOMED, CPT, LOINC, and other standard terminology systems — is a specialized skill that creates bottlenecks when clinicians need to define cohort criteria or annotate research data. The Medical Code Retrieval System allows users to describe a clinical concept in natural language and receive the relevant standard codes, their hierarchical relationships, and alternative coding options. The system accelerates cohort definition, clinical data annotation, and regulatory submission preparation.

AI-Driven Retrieval-Augmented Generation (RAG) for Clinical Trials

Developing clinical trial protocols requires synthesis of evidence from published literature, regulatory guidance, and institutional precedent. The RAG system indexes research literature, clinical guideline documents, and historical trial protocols in the platform's vector database, enabling researchers to ask protocol development questions and receive synthesized, evidence-grounded answers with source citations. This tool significantly accelerates protocol drafting, reduces literature review burden, and ensures protocol elements are grounded in current evidence.

Technology Stack

insert table

Conclusion

The Medical Data into Actionable Insights engagement demonstrates what is achievable when clinical domain expertise, advanced machine learning, and rigorous data engineering come together in a platform built for healthcare research.

Pfactorial delivered more than a data pipeline we delivered a research acceleration platform. By transforming siloed, heterogeneous clinical and genomic data into a secure, standardized, and analytically accessible resource, we enabled the research institution to conduct studies that were previously impractical, at a speed and scale that would have been impossible with conventional tools.

The custom PHI de-identification model that outperforms the NLM benchmark, the multi-omics processing pipelines, the AI tools that democratize data access for non-technical clinicians each component reflects Pfactorial's commitment to building solutions that are not just technically capable but genuinely transformative for the researchers and clinicians who use them.

This platform is a living system, designed to grow with the institution's data and research ambitions. As new patient data is ingested, new sequencing runs processed, and new research questions emerge, the architecture scales to meet them providing a durable foundation for the next decade of precision medicine research.