Back
July 02, 2024
ML Infra, Classifiers & RL
ANOMALY DETECTION IN CYBER SECURITY
Pfactorial
Share
Introduction
Cyber security is one of the most critical fields in the tech industry. It is a challenging task for cyber security companies to ensure the security of their client’s product ,as hackers continuously discover new vulnerabilities everyday to compromise a system. Due to the dynamic nature of vulnerabilities it is hard for rule-based software to detect the new threats. This is where machine learning comes to the rescue !
Instead of implementing a rule-based system we can employ a Machine learning model that generalizes the prediction of new vulnerabilities from trained data. In this blog we will be implementing a URL classification model that classifies URLs into phishing, defacement, malware and benign.
Before moving on to the specifics of the implementation ,let’s understand a little bit about the use cases of machine learning in cybersecurity , as well as its benefits and challenges.
When and why ?
Consider the scenario : You are running a successful ecommerce business with a large customer base .Due to the vast amount of data on your website, data scrapers are continually scrapping your site. You have database the stores the user behaviors including the number of requests per second, patterns of clicking at different UI elements on the website etc. A Machine learning model can be trained on this data and it can easily identify whether the user is a real one or an automated program.
Use Cases
Detecting bots - Machine learning models can be used to distinguish between a normal user and a bot that sends requests to a website. E-commerce companies can use these models for preventing unauthorized data scraping from their websites.
Malware detection - Malware detection can be done using Machine Learning models trained on large amounts of data. For instance ,Large language models can be fine tuned to identify threats and malwares in a company’s code base.
Phishing detection - Phishing attacks are one of the most used types of cyber attacks used to steal sensitive information from people using fake websites ,emails etc. Machine learning models can be used to detect phishing website URLs.
Code Analysis - To ensure that code is production ready ,it should have zero vulnerabilities. To analyze the code and detect vulnerabilities Machine learning models can be used.
Benefits
Adapting to new vulnerabilities : Since ML models can be trained on new data. Data about new vulnerabilities can be used to fine tune an existing vulnerability detection model.
Quick solutions : An ML model can learn new vulnerabilities by training , instead of writing new rules for each new vulnerability.
Real time analysis : Incoming network data can be analysed to cluster network traffic into different categories including scrappers ,normal users etc.
Challenges
Cost - Implementing an ML model and infrastructure for analyzing large amounts of network traffic requires significant financial investment.
Poisoning attacks - ML models depend on their training data .If a hacker can manipulate the training data ,he/she can manipulate the outputs of the model and can bypass the detection. This type of attacks are called Poisoning attacks
Input manipulation - Machine learning models are sensitive to their inputs
A small change in the input may change the predicted output
Implementing your own Model for detecting malicious URLs
Dataset
The dataset used for training our model can be downloaded from Kaggle.https://www.kaggle.com/datasets/sid321axn/malicious-urls-dataset
The dataset contains urls and their corresponding categories:
● Malware
● Benign
● Phishing
● Defacement
Malware URLs contain malwares which can be used to compromise a system when a user visits the URL.
Benign URLs are safe as they are normal URLs that we see regularly on the web.
Phishing URLs are URLs which mimic original URLs and try to steal sensitive information like credit card details.
Defacement URLs are URLs of websites which are already hacked and often hackers change their appearance.
Data format
Here we are not performing any data preprocessing steps ,since we want to preserve the structure of the URLs.
Creating a subset data frame
We are using only a small part of the data. We take data from each URL type equally and create a new data frame.
Encoding the labels
Since ML models take numerical input ,we need to numerically encode our models. Here we create a dictionary that maps each type to its corresponding numerical value and create a new column for the numerical labels.
Training
Before starting the training of the model we separate the dataset into features and labels. Used TfidfVectorizer to vectorize our data and splits the data into train and test sets.
Here we use the Random forest classifier to create our model with n_estimators 100.n_estimators define the number of decision trees we want to use in the Randomforest algorithm.
We train our model using the fit method. After comparing the predictions on the test data and the actual data ,here we can see that our model has an accuracy of 95.6%.
Saving the model
We use pickle to save our vectorizer and the model
Api
We have successfully created our model .Now it’s time to create an api for our model so that people can use it right !
Here we load our saved models and create a utility function predict that returns the category of an input URL. The function vectorizes the URL and passes it to the model to predict the category.
The url_category function is used to handle the post requests. The function accepts the URL as input and uses the predict function to return the outputs.
Testing the api
We test our model using the requests library and send a post request with the url.
Conclusion
The applications of machine learning in cybersecurity is not limited to the use of ML models. Advanced deep learning models are also used to detect anomalies. We have only covered a small part of the vast topic. Hope you learned something new from our blog. Happy learning !