Neural Computing and Applications Software
A framework for talk-to-me detection and human action recognition on smart eyewear
Abstract
Understanding who is talking to me and what actions are being performed represents an important aspect of human perception on embedded edge devices, such as smart eyewear and mobile robots. While Talk-To-Me (TTM) detection and skeleton-based Human Action Recognition (HAR) have typically been studied as separate tasks, in real-world scenarios, these tasks contribute complementary information toward understanding what is happening around the user. The outputs of these tasks are complementary: TTM indicates whether an interaction is directed toward the user, while HAR describes the ongoing physical actions of surrounding people. Exploiting them jointly is therefore beneficial for robust real-world interaction understanding. However, most existing TTM and HAR models are computationally demanding and poorly suited for deployment on resource-constrained edge devices. We propose the ExoAction, an edge-AI-compatible framework that jointly addresses TTM detection and HAR. For TTM detection, we develop EAIC-TTM, a lightweight RGB-only model specifically designed for edge-AI hardware. Despite using only visual input, EAIC-TTM achieves mAP comparable to the RGB+Audio Ego4D baseline while improving overall accuracy and requiring only RGB input, demonstrating an efficient solution for edge deployment. For HAR, we build upon HT-ConvNet, a compact convolutional model originally designed for efficient skeleton-sequence human action analysis. We enhance it with a new online recognition pipeline that processes incoming skeleton frames without needing the full action sequence. Our approach is evaluated on the POLIMI-ITW-S and NTU benchmarks. Comprehensive experiments demonstrate that our framework achieves competitive accuracy while maintaining high computational efficiency. Furthermore, deployment on a smart eyewear prototype equipped with an NVIDIA Jetson Orin Nano validated the framework’s real-world feasibility, achieving promising performance for pose estimation and HAR, and demonstrating strong potential for fully on-device egocentric perception in edge-AI applications.