Natural Language ProcessingData ScienceHealthcare & Clinical+4
NATURAL LANGUAGE PROCESSING FOR HEALTH RECORDS
Jun 27, 20247 min readRead
Back
June 27, 2024
ML Infra, Classifiers & RL
DATA CENTRIC AI
pfactorial
Share
Data is an important factor in machine learning. Quality data is required for an ML model to make accurate predictions. The machine learning paradigm that aims to systematically improve the dataset for better performance of the model is called Data centric AI. Another paradigm is Model centric AI where the aim is to improve the model for a given dataset. There are various methodologies used in Data centric AI for producing better quality data including outlier detection ,feature engineering ,feature selection, etc.
When and why?
- Data quality : High quality data is essential for training robust and accurate AI models.
- Avoiding bias : AI models trained on low quality data have higher chances of being biased . Employing data centric ai methods like outlier detection can remove bias from the data.
- Reproducibility : Data centric ai practices ensure that data is well documented and versioned
Data centric practices in Natural language processing (NLP)
In machine learning tasks like text classification,natural language processing techniques are used to ensure the data quality before training a machine learning model.
These techniques include:
● Removing special characters from the data
● Tokenizing the text
● Stopword removal .Removing common words like ‘the’, ’is’ etc.
● Data labelling
● Lemmatization converting words to their root form.
Implementing Data centric ai using NLP
Let’s implement a news classification model which classified news into business and not business categories.
Data loading
Loaded the data and assigned label 1 for news in the business category and 0 for non-business news.
Non-data centric text processing
Here we are not doing any nlp text preprocessing steps .We are simply removing empty strings from our data.
Splitting the data in to training and test sets
Split the data into two parts. One for training our model and the other for testing the trained model.
Vectorizing the data and dimensionality reduction
Before using the text data to train our model we need to convert the data to numbers.The ml algorithm can only take numbers as input.Here we use TfidfVectorizer to convert the text data into numbers.Since we are using news data for training, the vectors can be long and the model will need more computing power to train.To avoid this we reduce the dimensions of our data using Truncated Single value decomposition.Here we are reducing the size of the vectors to 200.For instance, if the size of a vector is 4000 it will be reduced to 200 .
Initializing the model
Here we are using a Support vector classifier for classifying news.Created two functions ,training_pipeline and testing_pipeline.The training pipeline is used to train the classifier and testing_pipeline is used to test the classifier. Since this is a binary classification problem SVC will try to find a margin that maximises the distance between the two classes.
The training pipeline converts the input data to vectors and then applies dimensionality reduction and trains and saves the model,vectorizer,truncated svd
Testing pipeline returns the predictions .
Training
Train the model by calling the function and pass the training data as argument.
Testing
Test the trained model using the testing pipeline and save the results in the y_pred variable.
Confusion matrix to understand model performance
In the confusion matrix we can see the number of actual Business news classified by the model as business news and number of actual non-business news classified by the model as non-business news, i.e, True positives and true negatives. Now let’s compare this with the confusion matrix of the model after data centric processing.
Data centric model creation
All of the above steps are the same except for the text processing function.
Let’s see what are the differences between non-data centric text processing and data- centric text processing.
Here we are removing the special characters from the data and replacing it with space to avoid noise from the dataset.
Removed stopwords from the text as these words are common and they usually don’t add any value in the model training process.
Lemmatization is used to convert words to their root form.This helps the model to generalise well on new data.
For example the word ‘walking’ will be converted to its root form ‘walk’.
Confusion matrix of data centric model
Here we can see that there is a slight increase in the number of True positives and True negatives,which means that our data centric text processing approach improved the model.
Here we load our saved model ,vectorizer and truncated svd.Created a function predict for returning the results of the input text to the model.Used the function in the flask api.
Conclusion
In this blog we have explored the data centric text preprocessing techniques for a text classification problem.The data centric model building approach is not only limited to text classification but it can be applied to many other data formats including numerical data,image data etc.
