Multimedia Systems Computer Networks and Communications

Training-free grounded attention decoupling for reliable multi-entity attribute binding in text-to-image diffusion

Chen, Yufei, Huang, Zhiliang, Zhang, Youyan
Springer Nature · 2026-11-14 · DOI 10.1007/s00530-026-02743-7

Abstract

Text-to-image diffusion models often struggle to bind distinct attributes to multiple entities, particularly when objects share the same semantic class or appear in crowded layouts. We present GSA-Diffusion, a training-free grounded attention decoupling framework for U-Net-based diffusion models. It derives semantic and spatial priors from entity-attribute parsing, instance masks, and structural guidance. Its Attention Decoupling and Injection mechanism addresses two complementary sources of attribute leakage. Mask-guided cross-attention injection assigns attribute information to the corresponding entity regions, while a topology-preserving self-attention bias restricts feature propagation between different entities and preserves entity-background interaction. A dynamic scheduling strategy gradually relaxes these constraints during later denoising stages, improving texture continuity, illumination, and boundary naturalness. Experiments on a multi-entity attribute-binding benchmark using SD1.5 and SDXL show improved binding accuracy and reference-mask similarity over optimization-based and attention-intervention baselines. Qualitative results demonstrate reduced attribute leakage and better structural preservation. Ablation studies confirm the distinct contributions of cross-attention injection, self-attention isolation, and dynamic scheduling in balancing semantic fidelity and perceptual realism. These results demonstrate that explicit attention decoupling provides a practical training-free solution for controllable compositional image generation.

Read at publisher →