Comparative Analysis of Traditional Machine Learning, Deep Learning, and Hybrid Ensemble Models for Anomaly Detection and Web Application Firewall Optimisation

Abstract

Anomaly detection is an important component of cybersecurity, particularly in safeguarding web application firewalls (WAFs) from malicious traffic. In this study, we perform a comparative analysis of three Machine Learning (ML) approaches: Random Forest (RF), Convolutional Neural Network (CNN), and a stacking ensemble combining RF and CNN with Logistic Regression (LR) as the meta-learner to explore the most effective approach for anomaly detection. To ensure a fair comparison, we trained all models under consistent preprocessing pipelines, including data class balancing using the SMOTE technique to address the common imbalance in attack data. The results of this study showed that the stacking ensemble outperformed the other models, achieving the highest accuracy (99.97%). The CNN model followed closely with comparable accuracy (99.94%), while also offering significant advantages in terms of computational efficiency and interpretability, particularly when supplemented with SHAP analysis. In contrast, the RF model achieved moderate accuracy (80.41%) but demonstrated strengths in interpretability and efficiency. These findings highlight that, with effective preprocessing, a standalone CNN can provide a practical and resource-efficient alternative to more complex ensemble models. The findings of this study highlight the importance of preprocessing in optimising model performance and propose CNN as a suitable solution for real-time cybersecurity applications. Future research should explore these models across diverse datasets, further investigate hybrid deep learning (DL) frameworks, and integrate advanced interpretability methods to enhance model transparency and trust in ML-based security systems.