Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model

Metamorphic malware modifies its code structure using a morphing engine to evade traditional signature-based detection. Previous research has shown the use of opcode instructions as feature representation with Hidden Markov Model in the context of metamorphic malware detection. However, it would be...

Full description

Bibliographic Details
Main Authors: Ling, Yeong Tyng, Mohd Sani, Nor Fazlida, Abdullah, Mohd Taufik, Abdul Hamid, Nor Asilah Wati
Format: Article
Published: Springer Cham 2021
Online Access:http://psasir.upm.edu.my/id/eprint/94169/
_version_ 1848861925812731904
author Ling, Yeong Tyng
Mohd Sani, Nor Fazlida
Abdullah, Mohd Taufik
Abdul Hamid, Nor Asilah Wati
author_facet Ling, Yeong Tyng
Mohd Sani, Nor Fazlida
Abdullah, Mohd Taufik
Abdul Hamid, Nor Asilah Wati
author_sort Ling, Yeong Tyng
building UPM Institutional Repository
collection Online Access
description Metamorphic malware modifies its code structure using a morphing engine to evade traditional signature-based detection. Previous research has shown the use of opcode instructions as feature representation with Hidden Markov Model in the context of metamorphic malware detection. However, it would be more feasible to extract a file feature at fine-grained level. In this paper, we propose a novel detection approach by generating structural features through computing a stream of byte chunks using compression ratio, entropy, Jaccard similarity coefficient and Chi-square statistic test. Nonnegative Matrix Factorization is also considered to reduce the feature dimensions. We then use the coefficient vectors from the reduced space to train Hidden Markov Model. Experimental results show there is different performance between malware detection and classification among the proposed structural features.
first_indexed 2025-11-15T13:08:53Z
format Article
id upm-94169
institution Universiti Putra Malaysia
institution_category Local University
last_indexed 2025-11-15T13:08:53Z
publishDate 2021
publisher Springer Cham
recordtype eprints
repository_type Digital Repository
spelling upm-941692023-03-29T01:30:43Z http://psasir.upm.edu.my/id/eprint/94169/ Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model Ling, Yeong Tyng Mohd Sani, Nor Fazlida Abdullah, Mohd Taufik Abdul Hamid, Nor Asilah Wati Metamorphic malware modifies its code structure using a morphing engine to evade traditional signature-based detection. Previous research has shown the use of opcode instructions as feature representation with Hidden Markov Model in the context of metamorphic malware detection. However, it would be more feasible to extract a file feature at fine-grained level. In this paper, we propose a novel detection approach by generating structural features through computing a stream of byte chunks using compression ratio, entropy, Jaccard similarity coefficient and Chi-square statistic test. Nonnegative Matrix Factorization is also considered to reduce the feature dimensions. We then use the coefficient vectors from the reduced space to train Hidden Markov Model. Experimental results show there is different performance between malware detection and classification among the proposed structural features. Springer Cham 2021-10-31 Article PeerReviewed Ling, Yeong Tyng and Mohd Sani, Nor Fazlida and Abdullah, Mohd Taufik and Abdul Hamid, Nor Asilah Wati (2021) Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model. Journal of Computer Virology and Hacking Techniques, 18. pp. 183-203. ISSN 2263-8733 https://link.springer.com/article/10.1007/s11416-021-00404-z 10.1007/s11416-021-00404-z
spellingShingle Ling, Yeong Tyng
Mohd Sani, Nor Fazlida
Abdullah, Mohd Taufik
Abdul Hamid, Nor Asilah Wati
Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model
title Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model
title_full Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model
title_fullStr Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model
title_full_unstemmed Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model
title_short Metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model
title_sort metamorphic malware detection using structural features and nonnegative matrix factorization with hidden markov model
url http://psasir.upm.edu.my/id/eprint/94169/
http://psasir.upm.edu.my/id/eprint/94169/
http://psasir.upm.edu.my/id/eprint/94169/