Journal of King Saud University Computer and Information Sciences General Computer Science

CandidateFusion-MKAN: a unified framework for robust multimodal crisis classification across diverse supervision and evidence conditions

Zequn Wang, Shanshan Li, Qingjie Liu, Zhian Pan, Guan Li
Springer Nature · 2026-11-01 · DOI 10.1007/s44443-026-01415-w

Abstract

Abstract Multimodal crisis classification faces uncertainties in real-world emergency settings: text and images may support the same or different humanitarian categories, provide complementary cues, or become missing or degraded. This study considers these conditions under a single-output classification constraint in which the model returns one Top-1 category. We propose CandidateFusion-MKAN, a unified framework for robust single-output multimodal crisis classification. Modality-supported candidate-set learning (MSCSL) preserves categories supported by text and image annotations; candidate-set likelihood (CSL) concentrates probability on the observed candidate set, while a bounded adaptive soft target (BAST) guides within-set allocation. Separately supervised text, image, and joint experts preserve modality-specific and cross-modal predictive paths, combined by reliability-adaptive routing. Experiments on the official CrisisMMD splits cover 973 label-consistent and 1,264 label-disagreement test instances, a 463-instance proxy-defined complementary-evidence subset, 43 clean, missing, and degraded evidence conditions, and event- and disaster-type-disjoint transfer tests. Under label-consistent supervision, CandidateFusion-MKAN achieves 92.60% accuracy. On the complete test set, it reduces Set-NLL by 0.048 and admissibility ECE from 0.214 to 0.047 relative to matched text–image probability averaging. On the proxy-defined complementary-evidence subset, it matches the best AnyAcc of 85.53% while reducing Set-NLL from 0.431 to 0.363. These results provide evidence for robust single-output classification across diverse supervision and evidence conditions.

Read at publisher →