A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features
As internet encryption has grown to safeguard users’ privacy, malware has evolved to leverage encryption protocols such as Transport Layer Security (TLS) to conceal its hazardous connections. The difficulty and impracticality of decrypting TLS network traffic before it reaches the Intrusion Detectio...
| Main Author: | |
|---|---|
| Format: | Thesis |
| Language: | English |
| Published: |
2022
|
| Subjects: | |
| Online Access: | http://eprints.usm.my/60044/ http://eprints.usm.my/60044/1/24%20Pages%20from%20KINAN%20KESHKEH.pdf |
| _version_ | 1848884335428501504 |
|---|---|
| author | Keshkeh, Kinan |
| author_facet | Keshkeh, Kinan |
| author_sort | Keshkeh, Kinan |
| building | USM Institutional Repository |
| collection | Online Access |
| description | As internet encryption has grown to safeguard users’ privacy, malware has evolved to leverage encryption protocols such as Transport Layer Security (TLS) to conceal its hazardous connections. The difficulty and impracticality of decrypting TLS network traffic before it reaches the Intrusion Detection System (IDS) has driven numerous research studies to focus on anomaly-based malware detection without decryption employing various features and Machine Learning (ML) algorithms. Nonetheless, several of these studies used flow features with low feature importance value and poor capability to distinguish malicious flows, such as the number of packets sent and received in a flow or its duration. Furthermore, the outliers and frequency-based flow feature transformations (FTT) applied to mitigate the poor flow feature have several flaws. This thesis proposes a TLS-based malware detection (TLSMalDetect) approach based on ML classification to address flow feature utilization limitations in related work. TLSMalDetect includes periodicity-independent entropy-based flow set (EFS) features produced by an FFT technique. The efficiency of EFS features is assessed in two ways: (1) by comparing them to the relevant related work’s features of outliers and flow using four feature importance methods, and (2) by analyzing the classification performance in the scenarios with and without EFS features. This study also investigates TLSMalDetect detection performance using seven ML classification algorithms and identifies the one with the highest accuracy. |
| first_indexed | 2025-11-15T19:05:04Z |
| format | Thesis |
| id | usm-60044 |
| institution | Universiti Sains Malaysia |
| institution_category | Local University |
| language | English |
| last_indexed | 2025-11-15T19:05:04Z |
| publishDate | 2022 |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | usm-600442024-03-04T01:20:45Z http://eprints.usm.my/60044/ A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features Keshkeh, Kinan QA75.5-76.95 Electronic computers. Computer science As internet encryption has grown to safeguard users’ privacy, malware has evolved to leverage encryption protocols such as Transport Layer Security (TLS) to conceal its hazardous connections. The difficulty and impracticality of decrypting TLS network traffic before it reaches the Intrusion Detection System (IDS) has driven numerous research studies to focus on anomaly-based malware detection without decryption employing various features and Machine Learning (ML) algorithms. Nonetheless, several of these studies used flow features with low feature importance value and poor capability to distinguish malicious flows, such as the number of packets sent and received in a flow or its duration. Furthermore, the outliers and frequency-based flow feature transformations (FTT) applied to mitigate the poor flow feature have several flaws. This thesis proposes a TLS-based malware detection (TLSMalDetect) approach based on ML classification to address flow feature utilization limitations in related work. TLSMalDetect includes periodicity-independent entropy-based flow set (EFS) features produced by an FFT technique. The efficiency of EFS features is assessed in two ways: (1) by comparing them to the relevant related work’s features of outliers and flow using four feature importance methods, and (2) by analyzing the classification performance in the scenarios with and without EFS features. This study also investigates TLSMalDetect detection performance using seven ML classification algorithms and identifies the one with the highest accuracy. 2022-11 Thesis NonPeerReviewed application/pdf en http://eprints.usm.my/60044/1/24%20Pages%20from%20KINAN%20KESHKEH.pdf Keshkeh, Kinan (2022) A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features. Masters thesis, Perpustakaan Hamzah Sendut. |
| spellingShingle | QA75.5-76.95 Electronic computers. Computer science Keshkeh, Kinan A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features |
| title | A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features |
| title_full | A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features |
| title_fullStr | A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features |
| title_full_unstemmed | A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features |
| title_short | A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features |
| title_sort | machine learning classification approach to detect tls-based malware using entropy-based flow set features |
| topic | QA75.5-76.95 Electronic computers. Computer science |
| url | http://eprints.usm.my/60044/ http://eprints.usm.my/60044/1/24%20Pages%20from%20KINAN%20KESHKEH.pdf |