Software Quality Journal Программное обеспечение
Quantifying SZZ-induced evaluation bias in just-in-time defect prediction
Аннотация
Just-in-time defect prediction identifies defect-inducing commits at commit time to focus quality assurance efforts, but its validity depends on the quality of underlying defect datasets, typically generated using SZZ-based algorithms. Because the correctness of these datasets is discussed frequently, this study sets out to quantify how residual SZZ errors propagate into just-in-time defect prediction evaluation. We construct a manually validated ground-truth sample covering 420 commits across 6 open-source projects and provide a stratified, weighted evaluation protocol for estimating population-level performance. Using this sample, we evaluate PR-SZZ-generated datasets and reassess state-of-the-art model performance. We find that PR-SZZ generates wrong defect labels for approximately 12.4% of commits, mainly through false positives. When just-in-time defect prediction models learned on these defect datasets are evaluated against manual ground truth rather than PR-SZZ labels, average model recall drops from 0.72 to 0.64 and precision from 0.27 to 0.19, indicating that SZZ-based evaluations can substantially misrepresent measured model performance. We also find that previously reported performance improvements gained by feature engineering only partly persist under manual evaluation, suggesting that added model capacity may capture systematic labeling errors rather than genuine defect-inducing behavior, particularly as feature characteristics can correlate with SZZ mechanisms. Overall, our findings highlight the need for reliable ground-truth inference and rigorous evaluation practices.