To enable automatic and accurate quantitative evaluation of hand function rehabilitation in stroke patients, this paper proposes a skeleton-based gesture recognition and evaluation method. First, the MediaPipe framework is used to extract hand keypoints and connect them to form a hand skeleton, converting traditional RGB video datasets into hand skeleton datasets. Then, a 3D convolutional neural network (C3D) model is employed to train on and recognize hand functional movements. Based on correct recognition, further evaluation is conducted using the dynamic time warping (DTW) algorithm. The DTW distance between the motions of the healthy and affected hands performing the same action is calculated, aligning both temporally and spatially to represent the similarity in action execution. Experiments establish optimal DTW thresholds for distinguishing different rehabilitation ratings for each action, which serve as the criteria for quantitative evaluation.Results show that using skeletal data instead of traditional video improves gesture recognition accuracy to 99.01% and reduces training time. With the DTW algorithm, automatic hand function rehabilitation assessment is achieved.
ZHU Shiyi
,
LU Xiaofeng
. Skeleton-Based Gesture Recognition and Rehabilitation Assessment for Stroke Patients[J]. Journal of Applied Sciences, 2025
, 43(5)
: 817
-827
.
DOI: 10.3969/j.issn.0255-8297.2025.05.009
[1] Song X, Chen S, Jia J, et al. Cellphone-based automated Fugl-Meyer assessment to evaluate upper extremity motor function after stroke [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27(10): 2186-2195.
[2] Li K, Li Z, Zeng H, et al. Control of newly-designed wearable robotic hand exoskeleton based on surface electromyographic signals [J]. Frontiers in Neurorobotics, 2021, 15: 711047.
[3] Chen X, Gong L, Wei L, et al. A wearable hand rehabilitation system with soft gloves [J]. IEEE Transactions on Industrial Informatics, 2020, 17(2): 943-952.
[4] Alimanova M, Borambayeva S, Kozhamzharova D, et al. Gamification of hand rehabilitation process using virtual reality tools: using leap motion for hand rehabilitation [C]//First IEEE International Conference on Robotic Computing (IRC), 2017: 336-339.
[5] Li C, Yang H, Cheng L, et al. Quantitative assessment of hand motor function for poststroke rehabilitation based on HAGCN and multimodality fusion [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 2032-2041.
[6] 徐胜, 刘志诚, 涂鑫涛, 等. 基于手势识别的手部康复智能评估算法研究[J]. 自动化仪表, 2021, 42(12): 23-27.Xu S, Liu Z C, Tu X T, et al. Research on intelligent evaluation algorithm of hand rehabilitation based on gesture recognition [J]. Automation Instrumentation, 2021, 42(12): 23-27. (in Chinese)
[7] Tran D, Bourdev L, Fergus R, et al. Learning spatiotemporal features with 3D convolutional networks [C]//IEEE International Conference on Computer Vision, 2015: 4489-4497.
[8] Duan H, Zhao Y, Chen K, et al. Revisiting skeleton-based action recognition [C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 2969-2978.
[9] Hatamimajoumerd E, Kakhaki P D, Huang X, et al. Challenges in video-based infant action recognition: a critical examination of the state of the art [C]//IEEE/CVF Winter Conference on Applications of Computer Vision, 2024: 21-30.
[10] Jin S, Xu L, Xu J, et al. Whole-body human pose estimation in the wild [C]//16th European Conference, Glasgow, UK, 2020: 196-214.
[11] 金彦亮, 吴筱. 基于双流自适应时空增强图卷积网络的手语识别[J]. 应用科学学报, 2024, 42(2): 189-199. Jin Y L, Wu X. Sign language recognition based on two-stream adaptive enhanced spatial temporal graph convolutional network [J]. Journal of Applied Sciences, 2024, 42(2): 189-199. (in Chinese)
[12] Rakthanmanon T, Campana B, Mueen A, et al. Searching and mining trillions of time series subsequences under dynamic time warping [C]//18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2012: 262-270.
[13] Li Y, Zhu Y, Wang Y, et al. Research on dance evaluation technology based on human posture recognition [C]//International Conference of Pioneering Computer Scientists, Engineers and Educators, 2023: 78-88.
[14] 周升儒, 陈志刚, 邓伊琴. 基于PoseC3D的网球动作识别及评价方法[J]. 计算机工程与科学, 2023, 45(1): 95-102. Zhou S R, Chen Z G, Deng Y Q. Tennis action recognition and evaluation method based on PoseC3D [J]. Computer Engineering & Science, 2023, 45(1): 95-102. (in Chinese)
[15] Fugl-Meyer A R, Jsk L, Leyman I L, et al. The post stroke hemiplegic patient. 1. a method for evaluation of physical performance [J]. Scandinavian Journal of Rehabilitation Medicine, 1975, 7(1): 13-31.
[16] Lugaresi C, Tang J, Nash H, et al. MediaPipe: a framework for building perception pipelines [DB/OL]. (2019-06-14) [2024-08-13]. https://arxiv.org/abs/1906.08172.
