Programming and Computer Software Software
Adaptive Methods and Tools for Shallow Packet Inspection in Anomaly Detection within Encrypted Network Traffic
Abstract
Abstract The widespread adoption of encrypted network traffic severely limits the applicability of deep packet inspection (DPI) in modern cloud infrastructures. This paper addresses the challenge of accurate and scalable anomaly detection using shallow packet inspection (SPI)—an approach that relies solely on metadata from packet headers at OSI layers 2–4, without accessing payload contents. We propose a lightweight, kernel-level SPI framework implemented as a driver for the Hyper-V Extensible Switch, named RuStatExt, which aggregates network traffic into flows and extracts statistical features with negligible performance overhead. To maximize detection quality, we introduce a modified greedy feature selection algorithm and a dynamic hyperparameter tuning strategy, which scales linearly with the number of monitored virtual machines (VMs). The methodology includes VM filtering, comparison of feature selection techniques, evaluation of unsupervised models, and assessment of detection quality using Precision, Recall, and F1 score. Practical validation is performed using synthetic L2–L4 attacks (SYN/UDP floods) and high-load network traffic over a 1-Gb/s link. Our experiments show that Isolation Forest, combined with the features selected by the proposed greedy algorithm and the dynamically tuned hyperparameters, achieves an F1 score of 0.78, which is nearly two times higher than the static configuration using all features. Crucially, the RuStatExt driver introduces no statistically significant degradation in network throughput for either TCP or UDP traffic.