Outlier Detection via Latent Space and Probabilistic Boundaries: A Convex Hull Approach
Аннотация
Effective outlier detection in high-dimensional, noisy, heterogeneous, imbalanced, and nonlinear datasets is critical for identifying economic risks, safety concerns, and valuable underlying processes across scientific and industrial fields. Autoencoders address these challenges by compressing data into a latent space and reconstructing it, under the assumption that large reconstruction errors indicate outliers. However, autoencoders do not account for the sensitivity of the latent space to minor changes in the data and remain susceptible to distance concentration, limiting the reproducibility of the outliers identified. This study presents the probabilistic convex hull (PCH) methodology, a data-driven approach that analyzes the latent space in three steps. First, PCH approximates the joint distribution of the latent features using Knuth’s rule, avoiding the bandwidth selection required by kernel density estimation. Second, it computes the convex hull of the dominant probability contour, imposing no assumption of continuity or geometric shape. Third, a dual thresholding strategy applies a Bonferroni-adjusted cutoff to reduce false positives and an alpha ( α ) cutoff to reduce false negatives. Across 25 synthetic datasets spanning varying noise regimes, PCH achieved the highest F1 score in 20 cases, outperforming local outlier factor, isolation forest, density-based spatial clustering of applications with noise, LatentOut, and reconstruction-error-based variational autoencoders. Remarkably, PCH requires only the significance level from the user. For a real oil and gas dataset, the Bonferroni-adjusted cutoff prioritized precision where false positives are costly, whereas the α cutoff prioritized recall where missing outliers are harmful.