Journal of King Saud University Computer and Information Sciences General Computer Science

AMCA-SGG: Adaptive multi-prototype learning and cross-modal alignment for unbiased scene graph generation

Jinhao Fan, Yuanhao Xi, Chuanping Hu, Ramin Yahyapour
Springer Nature · 2026-08-24 · DOI 10.1007/s44443-026-01206-3

Abstract

Abstract Scene graph generation (SGG) aims to detect objects and recognize their semantic relationships from images, serving as an important bridge between visual perception and high-level semantic understanding. Existing debiasing methods primarily address inter-class long-tailed bias, while paying insufficient attention to intra-class semantic heterogeneity of predicates. High-frequency predicates typically encompass multiple semantic sub-patterns, causing instances of the same predicate to scatter widely in the visual feature space and resulting in a fundamental misalignment between feature distributions and classification objectives. To address these issues, we propose AMCA, an unbiased SGG framework integrating adaptive multi-prototype learning with cross-modal alignment. Specifically, the Multi-Prototype Construction (MPC) module employs a distribution-aware prototype allocation strategy to quantify the semantic distributional complexity of each predicate and dynamically assign prototype counts, explicitly modeling intra-class semantic heterogeneity. Building on this, the Hybrid Relation Refinement Module (HRRM) embeds semantic prototypes into the visual relation feature refinement process via a prototype-augmented attention mechanism, and further employs Mamba to model global contextual dependencies among different relation representations, achieving deep fusion of semantic priors and visual features. Furthermore, the Visual-Text Alignment (VTA) module aligns the fused visual relation representations with CLIP text embeddings, significantly enhancing the model’s ability to distinguish fine-grained semantics. Extensive experiments on Visual Genome and GQA demonstrate that AMCA achieves consistently competitive performance across multiple SGG tasks, with particularly strong improvements on the unbiased mR@K metric. Our code is publicly available at https://github.com/SJVision/AMCA .

Read at publisher →