Multimedia Systems Computer Networks and Communications

MemSAS: prototype-driven spatio-temporal memory with task-adaptive decoupled features for skeleton-based action segmentation

Sun, Jiakang, Peng, Cheng, Zhong, Yong, Tang, Yichuan, Li, Ke
Springer Nature · 2026-11-20 · DOI 10.1007/s00530-026-02800-1

Abstract

Skeleton-based temporal action segmentation aims to perform fine-grained action label prediction for each frame on long untrimmed skeleton sequences. However, existing methods often overlook the variations in spatial joint relationships across different action snippets, as well as the semantic correlations among action snippets of the same category, limiting their ability to learn discriminative representations for accurate action segmentation. To address these limitations, we propose MemSAS, a prototype-driven spatiotemporal memory framework for skeleton-based action segmentation. Specifically, the Spatio-Temporal Aware Dynamic Graph module constructs adaptive graphs to capture snippet-specific spatial dependencies, while the Spatial Expert Memory Attention module interacts with spatial motion prototypes to enhance discriminative joint representations. The Temporal Memory Attention module further models semantic correlations among action snippets through prototype-based similarity matching. Additionally, to address the feature coupling issue between action classification and boundary prediction tasks, we introduce a task-adaptive feature decoupling mechanism that employs a curvature-guided multi-scale TCN and a motion enhancement module to extract complementary features for action classification and boundary prediction. Extensive experiments on multiple public datasets demonstrate the effectiveness of MemSAS. Code is available at https://github.com/sjkucas/MemSAS .

Read at publisher →