Hybrid Detection System for Polymorphic Botnet Software Based on Invariant Features Analysis
DOI:
https://doi.org/10.32515/2664-262X.2026.13(44).41-51Keywords:
cybersecurity, botnets, polymorphic malware, hybrid analysis, machine learning, invariant features, Control Flow Graphs, anomaly detectionAbstract
The purpose of this article is to address the critical cybersecurity challenge of detecting modern polymorphic botnets operating within distributed networks and IoT infrastructures. Traditional signature-based protection mechanisms have become fundamentally ineffective against advanced malware that continuously mutates its binary structure and cryptographic hashes. This research aims to overcome these limitations by proposing a conceptual model and software implementation of a hybrid detection system focused exclusively on identifying invariant structural and behavioral features of malicious code.
To achieve this goal, the study formalizes a mathematical and structural-functional model of a polymorphic agent, strictly separating its variable components (such as dynamic decrypters and junk code) from its invariant execution logic. Based on this formalization, a client-server hybrid architecture was developed. The proposed system pipeline integrates a static analysis module that performs lightweight endpoint filtering by examining Shannon entropy, Control Flow Graph (CFG) topology, and OpCode n-gram distributions without executing the code. For suspicious objects that bypass initial static filters, a dynamic analysis module is engaged within a server-side hardened sandbox environment. This module monitors specific sequences of critical system API calls (e.g., memory injection patterns) and network activity while employing countermeasures against Anti-VM and evasion techniques. Finally, the extracted data from both analytical modules are normalized, aggregated into a unified feature vector, and processed by an ensemble machine learning classifier utilizing the Random Forest algorithm.
Experimental validation was conducted using a synthetically generated dataset comprising over 10,000 unique polymorphic malware samples alongside benign software files. The results demonstrate that the proposed hybrid approach achieves a detection accuracy exceeding 95% while successfully maintaining a false positive rate below 1%. Ultimately, the implemented system provides an optimal balance between the high resource consumption of pure heuristic sandboxes and the low efficacy of static scanners, offering a robust defense framework for corporate cybersecurity perimeters.
References
Список літератури
1. Al-Shurbaji T., et al. Deep learning-based intrusion detection system for detecting IoT botnet attacks: a review. IEEE Access. 2025. Vol. 13. P. 11792-11822.
2. Hejazi S. M., et al. A lightweight hybrid deep learning-based intrusion detection system for detecting botnet attacks in IoT networks. Journal of Scientific Research and Reports. 2025. Vol. 31, № 11. P. 97-120.
3. Lysenko S., Savenko O., Bobrovnikova K. DDoS Botnet Detection Technique Based on the Use of the Semi-Supervised Fuzzy c-Means Clustering. ICTERI Workshops. 2018. P. 688-695.
4. Avhankar M. S., Pawar J., Kumbhar V. A Comprehensive Survey on Polymorphic Malware Analysis: Challenges, Techniques, and Future Directions. Communications on Applied Nonlinear Analysis. 2025. Vol. 32, № 9. P. 2765-2776.
5. Kashtalian A., et al. Multi-computer malware detection systems with metamorphic functionality. Radioelectronic and Computer Systems. 2024. № 1. P. 152-175.
6. Berrios S., et al. Systematic review: Malware detection and classification in cybersecurity. Applied Sciences. 2025. Vol. 15, № 14. Art. 7747.
7. Miraoui M., Belgacem M. B. Binary and multiclass malware classification of windows portable executable using classic machine learning and deep learning. Frontiers in Computer Science. 2025. Vol. 7. Art. 1539519.
8. Kamdan, et al. Static Malware Detection and Classification Using Machine Learning: A Random Forest Approach. Engineering Proceedings. 2025. Vol. 107, № 1. Art. 76.
9. Aryal K., et al. A survey on adversarial attacks for malware analysis. IEEE Access. 2024. Vol. 13. P. 428-459.
10. Fellicious C., et al. Malware Detection based on API calls. arXiv preprint. 2025. arXiv:2502.12863.
11. Zhang S., et al. A malware-detection method using deep learning to fully extract API sequence features. Electronics. 2025. Vol. 14, № 1. Art. 167.
12. Ferdous J., et al. A survey on ml techniques for multi-platform malware detection: Securing pc, mobile devices, iot, and cloud environments. Sensors. 2025. Vol. 25, № 4. Art. 1153.
13. Al-Ghanem W., et al. MAD-ANET: malware detection using Attention-Based deep neural networks. Computer Modeling in Engineering & Sciences. 2025. Vol. 143, № 1. P. 1009.
14. Ali M., et al. Botnet detection in internet of things using stacked ensemble learning model. Scientific Reports. 2025. Vol. 15, № 1. Art. 21012.
15. Dustova S., et al. AI-powered malware detection: a hybrid CNN-RNN model for real-time threat analysis. Optical and Computational Technologies for Measurements and Industrial Applications (OptiComp 2025). 2025. Vol. 13803. P. 711-716.
16. Feng P., et al. DawnGNN: Documentation augmented windows malware detection using graph neural network. Computers & Security. 2024. Vol. 140. Art. 103788.
17. Garg U., Kumar S., Kumar M. IHBOT: An Intelligent and Hybrid Model for Investigation and Classification of IoT Botnet. International Journal of Computer Network and Information Security. 2024. Vol. 16, № 5. P. 108-118.
18. Guo W., et al. MalHAPGNN: An enhanced call graph-based malware detection framework using hierarchical attention pooling graph neural network. Sensors. 2025. Vol. 25, № 2. Art. 374.
19. Javed A., et al. Adamw+: machine learning framework to detect domain generation algorithms for malware. IEEE Access. 2024. Vol. 12. P. 79138-79150.
20. Khurana P. Malware Detection in IoT Devices Using Machine Learning: A Review. Proc. 2024 International Conference on Computational Intelligence and Computing Applications (ICCICA). 2024. P. 203-209.
21. Kulkarni P., O'Shaughnessy S. Malware Detection Using Dynamic Graph Neural Networks. Proc. European Conference on Cyber Warfare and Security (ECCWS). 2025. P. 830-837.
22. Markowsky G., et al. The Technique for Metamorphic Viruses' Detection Based on Its Obfuscation Features Analysis. ICTERI workshops. 2018. P. 680-687.
23. Pomorova O., et al. Metamorphic Viruses Detection Technique Based on the the Modified Emulators. ICTERI. 2016. P. 375-383.
24. Qu T., et al. Demystifying Feature Engineering in Malware Analysis of API Call Sequences. arXiv preprint. 2025. arXiv:2512.01666.
25. Ramskyi I., et al. System for cybersecurity evaluation of corporate networks. Computer systems and information technologies. 2025. № 2. P. 123-131.
26. Savenko O., et al. Approach for the unknown metamorphic virus detection. Proc. 9th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems (IDAACS). 2017. Vol. 1. P. 71-76.
27. Savenko O., et al. Metamorphic Viruses' Detection Technique Based on the Equivalent Functional Block Search. ICTERI. 2017. P. 555-568.
28. Shokouhinejad H., et al. Explainable Attention-Guided Stacked Graph Neural Networks for Malware Detection. arXiv preprint. 2025. arXiv:2508.09801.
29. Youssef N., Elbaraway N., Elmaghraby A. Transformer-Based API Call Sequence Modeling for Dynamic Malware Detection. Proc. SoutheastCon 2025. 2025. P. 494-500.
30. Zakaria M., et al. Obfuscated file-less malware detection using integrating memory forensics data with machine learning techniques. Applied Computing and Informatics. 2025. P. 1-16.
References
1. Al-Shurbaji, T., Anbar, M., Manickam, S., Hasbullah, I. H., Alfriehat, N., Alabsi, B. A., ... & Hashim, H. (2025). Deep learning-based intrusion detection system for detecting IoT botnet attacks: a review. IEEE Access, 13, 11792-11822.
2. Hejazi, S. M., Alshalabi, A. Y., Hatamleh, M., & Albaroudi, E. (2025). A lightweight hybrid deep learning-based intrusion detection system for detecting botnet attacks in IoT networks. Journal of Scientific Research and Reports, 31(11), 97-120.
3. Lysenko, S., Savenko, O., & Bobrovnikova, K. (2018). DDoS Botnet Detection Technique Based on the Use of the Semi-Supervised Fuzzy c-Means Clustering. In Proceedings of the ICTERI Workshops (pp. 688-695).
4. Avhankar, M. S., Pawar, J., & Kumbhar, V. (2025). A Comprehensive Survey on Polymorphic Malware Analysis: Challenges, Techniques, and Future Directions. Communications on Applied Nonlinear Analysis, 32(9), 2765-2776.
5. Kashtalian, A., Lysenko, S., Savenko, O., Nicheporuk, A., Sochor, T., & Avsiyevych, V. (2024). Multi-computer malware detection systems with metamorphic functionality. Radioelectronic and Computer Systems, (1), 152-175.
6. Berrios, S., Leiva, D., Olivares, B., Allende-Cid, H., & Hermosilla, P. (2025). Systematic review: Malware detection and classification in cybersecurity. Applied Sciences, 15(14), 7747.
7. Miraoui, M., & Belgacem, M. B. (2025). Binary and multiclass malware classification of windows portable executable using classic machine learning and deep learning. Frontiers in Computer Science, 7, 1539519.
8. Kamdan, Pratama, Y., Munzi, R. S., Mustafa, A. B., & Kharisma, I. L. (2025). Static Malware Detection and Classification Using Machine Learning: A Random Forest Approach. Engineering Proceedings, 107(1), 76.
9. Aryal, K., Gupta, M., Abdelsalam, M., Kunwar, P., & Thuraisingham, B. (2024). A survey on adversarial attacks for malware analysis. IEEE Access, 13, 428-459.
10. Fellicious, C., Bischof, M., Mayer, K., Eikenberg, D., Hausotte, S., Reiser, H. P., & Granitzer, M. (2025). Malware Detection based on API calls. arXiv preprint arXiv:2502.12863.
11. Zhang, S., Gao, M., Wang, L., Xu, S., Shao, W., & Kuang, R. (2025). A malware-detection method using deep learning to fully extract API sequence features. Electronics, 14(1), 167.
12. Ferdous, J., Islam, R., Mahboubi, A., & Islam, M. Z. (2025). A survey on ml techniques for multi-platform malware detection: Securing pc, mobile devices, iot, and cloud environments. Sensors, 25(4), 1153.
13. Al-Ghanem, W., Ul, E., Zia, T., Faheem, M., Imran, M., & Ahmad, I. (2025). MAD-ANET: malware detection using Attention-Based deep neural networks. Computer Modeling in Engineering & Sciences, 143(1), 1009.
14. Ali, M., Mushtaq, M. F., Akram, U., Aray, D. G., Vergara, M. M., Karamti, H., & Ashraf, I. (2025). Botnet detection in internet of things using stacked ensemble learning model. Scientific Reports, 15(1), 21012.
15. Dustova, S., Abuzalova, M., Adizova, N., Ruzieva, D., & Ruziev, Y. (2025). AI-powered malware detection: a hybrid CNN-RNN model for real-time threat analysis. In Optical and Computational Technologies for Measurements and Industrial Applications (OptiComp 2025) (Vol. 13803, pp. 711-716).
16. Feng, P., Gai, L., Yang, L., Wang, Q., Li, T., Xi, N., & Ma, J. (2024). DawnGNN: Documentation augmented windows malware detection using graph neural network. Computers & Security, 140, 103788.
17. Garg, U., Kumar, S., & Kumar, M. (2024). IHBOT: An Intelligent and Hybrid Model for Investigation and Classification of IoT Botnet. International Journal of Computer Network and Information Security, 16(5), 108-118.
18. Guo, W., Du, W., Yang, X., Xue, J., Wang, Y., Han, W., & Hu, J. (2025). MalHAPGNN: An enhanced call graph-based malware detection framework using hierarchical attention pooling graph neural network. Sensors, 25(2), 374.
19. Javed, A., Rashid, I., Tahir, S., Saeed, S., Almuhaideb, A. M., & Alissa, K. (2024). Adamw+: machine learning framework to detect domain generation algorithms for malware. IEEE Access, 12, 79138-79150.
20. Khurana, P. (2024). Malware Detection in IoT Devices Using Machine Learning: A Review. In 2024 International Conference on Computational Intelligence and Computing Applications (ICCICA) (pp. 203-209). IEEE.
21. Kulkarni, P., & O'Shaughnessy, S. (2025). Malware Detection Using Dynamic Graph Neural Networks. In Proceedings of the European Conference on Cyber Warfare and Security (pp. 830-837).
22. Markowsky, G., Savenko, O., Lysenko, S., & Nicheporuk, A. (2018). The Technique for Metamorphic Viruses' Detection Based on Its Obfuscation Features Analysis. ICTERI workshops, 680-687.
23. Pomorova, O., Savenko, O., Lysenko, S., & Nicheporuk, A. (2016). Metamorphic Viruses Detection Technique Based on the the Modified Emulators. ICTERI, 375-383.
24. Qu, T., Zhu, H., Sun, L., Wang, H., Fei, H., He, Z., & Li, Z. (2025). Demystifying Feature Engineering in Malware Analysis of API Call Sequences. arXiv preprint arXiv:2512.01666.
25. Ramskyi, I., Drozd, A., Lyhun, O., & Ponochovna, O. (2025). System for cybersecurity evaluation of corporate networks. Computer systems and information technologies, (2), 123-131.
26. Savenko, O., Lysenko, S., Nicheporuk, A., & Savenko, B. (2017). Approach for the unknown metamorphic virus detection. In 2017 9th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS) (Vol. 1, pp. 71-76). IEEE.
27. Savenko, O., Lysenko, S., Nicheporuk, A., & Savenko, B. (2017). Metamorphic Viruses' Detection Technique Based on the Equivalent Functional Block Search. ICTERI, 555-568.
28. Shokouhinejad, H., Razavi-Far, R., Higgins, G., & Ghorbani, A. A. (2025). Explainable Attention-Guided Stacked Graph Neural Networks for Malware Detection. arXiv preprint arXiv:2508.09801.
29. Youssef, N., Elbaraway, N., & Elmaghraby, A. (2025). Transformer-Based API Call Sequence Modeling for Dynamic Malware Detection. In SoutheastCon 2025 (pp. 494-500). IEEE.
30. Zakaria, M., Mohamed, M. S., Hussein, S., & Salama, G. I. (2025). Obfuscated file-less malware detection using integrating memory forensics data with machine learning techniques. Applied Computing and Informatics, 1-16.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Andrii Holovatiuk, Bohdan Savenko
This work is licensed under a Creative Commons Attribution 4.0 International License.
