Real-time dynamic hand gesture recognition system on edge devices

Abstract

This paper presents a novel real-time dynamic hand gesture recognition system designed for efficient interaction with smart devices and touchless interfaces, with a focus on edge devices. The proposed system integrates Google Mediapipe for hand pose detection, which is lightweight enough to run on mobile devices, with a modified version of the DD-Net architecture, optimized for online classification of gestures using 2D and 3D data. Key innovations include the introduction of an auxiliary classification head to address class imbalance and an attention mechanism to improve the recognition of partially observed gestures. The system is evaluated on the NVGesture and SHREC22 datasets, achieving an accuracy of 0.784 and 0.924, respectively, surpassing previous benchmarks. Experimental results demonstrate the high efficiency of the proposed approach in real-time gesture recognition tasks, particularly on mobile platforms. It is shown that the use of 3D data and an attention mechanism significantly improves recognition accuracy, especially for complex and partially visible gestures. The proposed system can be used in a wide range of applications, including virtual and augmented reality systems, robotics, and touchless interfaces on edge devices