LLMData ScienceRecruiting & HR Tech+4
LLMs to extract structured data from text
Jul 01, 20249 min readRead
Back
June 27, 2024
ML Infra, Classifiers & RL
NATURAL LANGUAGE PROCESSING FOR HEALTH RECORDS
Pfactorial
Share
Introduction
Natural language processing is one of the most useful field in Machine learning with a wide variety of applications including sentiment analysis, document classification etc. Healthcare is one of the fields where we can find a lot of applications of NLP. For instance a model trained in keyword extraction can extract important keywords from a patient’s medical history. In this blog we are going to implement a model for classifying between results and conclusions of a clinical trial dataset. The dataset contains results and conclusions of different experiments in medical field. We will be using spacy library in python for implementing the model.
Implementing an NLP model for text classification
Let’s start building our Text classification model in spacy for classifying clinical experiment data into results and conclusions.
Data preprocessing
Import the following libraries and Create an empty language pipeline. Since our dataset is in English use ’en’.
We will use this pipeline later for text preprocessing and saving our data in spacy’s binary format.
Dataset format : Our dataset consists of two columns one for the text data and one for its corresponding labels(conclusions and results).
Since our data is in string format we need to numerically encode it for training the model. Assign 1 for conclusions and 0 for results. Ml models need data in numerical format.
Removing special characters and stop words. Stop words are removed ,because they are not important for a classification task as words like ‘is’,’the’ etc. won’t help the model for classification. But remember one thing it is not always ideal to remove stop words for training an NLP model.
After removing the special characters and stop words, we store our cleaned text in a new column. Since we are dealing with we are not performing steps like lemmatization for converting the words to its root form as we want to preserve the context of the sentence.
Spitting the data into train, test and evaluation sets. Here the dev df is used for evaluation.
For training our model we need to modify our dataset to allow iteration. For that we will convert the data frames to tuple format and then convert the data to spacy binary format. The binary format of data is required for training.
so Convert the train, test and validation sets into binary format .Here we also assign true and False values for each record in the data .
Creating a configuration file
When we need to train our data on a model architecture provided by spacy a configuration file is needed.We can create the configuration file by running the following command in terminal:
Training the model
spacy.TextCatBOW.V2 is a bag of words model.
In simple terms Bag of Word models are models which classify text based on the presence or absence of known words to the model. The intuition behind bag of words is that documents are similar if they have similar content.
For training our model ,run the following command in the terminal :
API for the model
At this point we have successfully created our model and now we need to create an API for the model so that if we host this api on a server, anyone can use our trained model.
Implementing API using Flask
Here the preprocess function is the same function we used in the data preprocessing part . The get_category function takes the data from the post request and passes it to the model. The model classifies the input text into results and conclusions and returns the output.
then test the api and here we can see that the model classified the text as conclusions
Use Cases of NLP in healthcare
- Patient data tracking : It is really hard for a doctor to memorise a patient’s history at a hospital ,especially if the patient has a long history of consultation with the doctor. NLP techniques can be applied for identifying the most frequently used words in the patient's history and helps the doctor to get an overall understanding about the patient.
- Healthcare record translation : Hospitals can use language translation models to translate health records in a different language. This is useful when dealing with international patients.
- Text summarization : Summarising patient history data can be really useful during emergency situations to make quick decisions.
- Disease prediction : Disease prediction can be implemented using NLP by using a model that is trained in the patient's lifestyle habits ,food choices etc.
- Clinical trial matching : Selecting candidates for clinical trials by checking the suitable candidates with preferred medical conditions can be implemented using NLP.
Benefits
- Preventing adverse events : Analysing patient history and identifying adverse events sooner and helping prevent them from happening.
- Avoiding incorrect prescriptions : An NLP model trained on patient conditions and medications prescribed can be used to verify prescriptions and rectify errors if present.
- Speech to text note taking : NLP techniques can be used to create prescriptions by understanding the conversation between doctor and a patient.
- Reducing patient wait time : Implementing NLP techniques in prescriptions and patient history summarisation can reduce consultation time .
Challenges
- Medical Terminology - The medical terminology used in health care records may be domain specific and requires a large amount of such data for an NLP model to produce good results.
- Data privacy - Since medical data is highly sensitive and vulnerable to misuse, constant data monitoring is needed throughout the entire NLP model creation process.
- Context - A particular medical condition may have different causes for different patients and medication may change accordingly. If an NLP model is not provided with context during training it may give incorrect results .
- Interpretability : Since it is not always possible to know what factors lead to a particular prediction by an NLP model, the trustworthiness of NLP models is a matter of concern.
Future
By 2025 the estimated market size of NLP is expected to reach USD 3.7 billion. Consequently, more research and advanced NLP models are expected to be released. This also motivates the healthcare industry to adopt NLP models.
Conclusion
In this blog we have explored how to implement an NLP model using spacy for Classifying clinical trials data into results and conclusions. There are many more advanced concepts in spacy which you can explore by referring to the spacy official documentation : spaCy 101: Everything you need to know · spaCy Usage Documentation
