为实现脑卒中患者手功能康复情况的自动、精准定量评估，本文提出一种基于手部骨骼的手势识别与功能评估方法。首先，利用MediaPipe框架提取手部关键点并连接形成手骨骼模型，将传统的RGB视频数据集转化为手骨骼数据集。然后，通过C3D模型进行训练，实现手功能动作的识别。最后，在正确识别的基础上进一步评估，采用动态时间规整（dynamic time warping,DTW）算法，在实现时序对齐的同时引入空间对齐机制，通过计算患者健侧手与患侧手完成同一动作的DTW距离，量化动作执行的相似度，为每个动作找到最佳阈值作为定量评估的标准。实验结果表明，用骨骼数据代替传统视频数据，使手势识别的准确率提升至99.01%，缩短了训练时间，并结合DTW算法，实现了手功能康复情况的自动评估。
To enable automatic and accurate quantitative evaluation of hand function rehabilitation in stroke patients, this paper proposes a skeleton-based gesture recognition and evaluation method. First, the MediaPipe framework is used to extract hand keypoints and connect them to form a hand skeleton, converting traditional RGB video datasets into hand skeleton datasets. Then, a 3D convolutional neural network (C3D) model is employed to train on and recognize hand functional movements. Based on correct recognition, further evaluation is conducted using the dynamic time warping (DTW) algorithm. The DTW distance between the motions of the healthy and affected hands performing the same action is calculated, aligning both temporally and spatially to represent the similarity in action execution. Experiments establish optimal DTW thresholds for distinguishing different rehabilitation ratings for each action, which serve as the criteria for quantitative evaluation.Results show that using skeletal data instead of traditional video improves gesture recognition accuracy to 99.01% and reduces training time. With the DTW algorithm, automatic hand function rehabilitation assessment is achieved.
[1] Song X, Chen S, Jia J, et al. Cellphone-based automated Fugl-Meyer assessment to evaluate upper extremity motor function after stroke [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27(10): 2186-2195.
[2] Li K, Li Z, Zeng H, et al. Control of newly-designed wearable robotic hand exoskeleton based on surface electromyographic signals [J]. Frontiers in Neurorobotics, 2021, 15: 711047.
[3] Chen X, Gong L, Wei L, et al. A wearable hand rehabilitation system with soft gloves [J]. IEEE Transactions on Industrial Informatics, 2020, 17(2): 943-952.
[4] Alimanova M, Borambayeva S, Kozhamzharova D, et al. Gamification of hand rehabilitation process using virtual reality tools: using leap motion for hand rehabilitation [C]//First IEEE International Conference on Robotic Computing (IRC), 2017: 336-339.
[5] Li C, Yang H, Cheng L, et al. Quantitative assessment of hand motor function for poststroke rehabilitation based on HAGCN and multimodality fusion [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 2032-2041.
[6] 徐胜, 刘志诚, 涂鑫涛, 等. 基于手势识别的手部康复智能评估算法研究[J]. 自动化仪表, 2021, 42(12): 23-27.Xu S, Liu Z C, Tu X T, et al. Research on intelligent evaluation algorithm of hand rehabilitation based on gesture recognition [J]. Automation Instrumentation, 2021, 42(12): 23-27. (in Chinese)
[7] Tran D, Bourdev L, Fergus R, et al. Learning spatiotemporal features with 3D convolutional networks [C]//IEEE International Conference on Computer Vision, 2015: 4489-4497.
[8] Duan H, Zhao Y, Chen K, et al. Revisiting skeleton-based action recognition [C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 2969-2978.
[9] Hatamimajoumerd E, Kakhaki P D, Huang X, et al. Challenges in video-based infant action recognition: a critical examination of the state of the art [C]//IEEE/CVF Winter Conference on Applications of Computer Vision, 2024: 21-30.
[10] Jin S, Xu L, Xu J, et al. Whole-body human pose estimation in the wild [C]//16th European Conference, Glasgow, UK, 2020: 196-214.
[11] 金彦亮, 吴筱. 基于双流自适应时空增强图卷积网络的手语识别[J]. 应用科学学报, 2024, 42(2): 189-199. Jin Y L, Wu X. Sign language recognition based on two-stream adaptive enhanced spatial temporal graph convolutional network [J]. Journal of Applied Sciences, 2024, 42(2): 189-199. (in Chinese)
[12] Rakthanmanon T, Campana B, Mueen A, et al. Searching and mining trillions of time series subsequences under dynamic time warping [C]//18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2012: 262-270.
[13] Li Y, Zhu Y, Wang Y, et al. Research on dance evaluation technology based on human posture recognition [C]//International Conference of Pioneering Computer Scientists, Engineers and Educators, 2023: 78-88.
[14] 周升儒, 陈志刚, 邓伊琴. 基于PoseC3D的网球动作识别及评价方法[J]. 计算机工程与科学, 2023, 45(1): 95-102. Zhou S R, Chen Z G, Deng Y Q. Tennis action recognition and evaluation method based on PoseC3D [J]. Computer Engineering & Science, 2023, 45(1): 95-102. (in Chinese)
[15] Fugl-Meyer A R, Jsk L, Leyman I L, et al. The post stroke hemiplegic patient. 1. a method for evaluation of physical performance [J]. Scandinavian Journal of Rehabilitation Medicine, 1975, 7(1): 13-31.
[16] Lugaresi C, Tang J, Nash H, et al. MediaPipe: a framework for building perception pipelines [DB/OL]. (2019-06-14) [2024-08-13]. https://arxiv.org/abs/1906.08172.
