Pattern Analysis and Applications Computer Vision and Pattern Recognition

PersonaDiff: diffusion-based style control for speech-synchronized 3D facial animation

Li, Wei, Song, Yupu, Liu, Yazhi
Springer Nature · 2026-11-02 · DOI 10.1007/s10044-026-01799-5

Abstract

Creating realistic 3D facial animations from speech is essential for digital humans and high-fidelity content creation. A major challenge is achieving accurate lip synchronization while modeling speaker-dependent dynamic facial styles. We propose PersonaDiff, an end-to-end diffusion-based framework for speech-driven 3D facial animation with explicit dynamic style modeling. Specifically, a motion encoder extracts fine-grained facial motion features from reference facial sequences, which are fused with audio-derived emotion features to construct a dynamic style representation for conditional generation. An enhanced decoder combining multi-head rotary attention with a learnable motion amplification module further improves temporal coherence and enhances subtle facial motions during the denoising process. Extensive experiments on five diverse datasets demonstrate that PersonaDiff consistently outperforms state-of-the-art methods in reconstruction accuracy, lip synchronization, facial motion dynamics, and perceptual quality, achieving up to 18.0% lower lip vertex error and 61.6% lower facial dynamics deviation than competing approaches. Qualitative results further show that the proposed dynamic style representation enables the generation of diverse speaker-dependent facial motions under different speaking styles. Overall, PersonaDiff provides an effective framework for personalized speech-driven 3D facial animation and is well suited for applications such as virtual avatars, digital content creation, and other digital human systems. Our source code is available at: https://github.com/Andy-spe/PersonaDiff/tree/master .

Read at publisher →