English

应用科学学报 ›› 2025, Vol. 43 ›› Issue (5): 817-827.doi: 10.3969/j.issn.0255-8297.2025.05.009

• 信号与信息处理 • 上一篇    

基于骨骼的脑卒中患者手势识别与康复评估

朱诗逸, 陆小锋   

  1. 上海大学 通信与信息工程学院, 上海 200444
  • 收稿日期:2024-08-13 发布日期:2025-10-16
  • 通信作者: 陆小锋，高级工程师，研究方向为智能信息处理。E-mail:luxiaofeng@shu.edu.cn E-mail:luxiaofeng@shu.edu.cn
  • 基金资助:
    上海市科委科技创新行动计划（No. 22511103304, No. 22511103403）

Skeleton-Based Gesture Recognition and Rehabilitation Assessment for Stroke Patients

ZHU Shiyi, LU Xiaofeng   

  1. School of Communication and Information Engineering, Shanghai University, Shanghai 200444, China
  • Received:2024-08-13 Published:2025-10-16

PDF

18

摘要： 为实现脑卒中患者手功能康复情况的自动、精准定量评估，本文提出一种基于手部骨骼的手势识别与功能评估方法。首先，利用MediaPipe框架提取手部关键点并连接形成手骨骼模型，将传统的RGB视频数据集转化为手骨骼数据集。然后，通过C3D模型进行训练，实现手功能动作的识别。最后，在正确识别的基础上进一步评估，采用动态时间规整（dynamic time warping,DTW）算法，在实现时序对齐的同时引入空间对齐机制，通过计算患者健侧手与患侧手完成同一动作的DTW距离，量化动作执行的相似度，为每个动作找到最佳阈值作为定量评估的标准。实验结果表明，用骨骼数据代替传统视频数据，使手势识别的准确率提升至99.01%，缩短了训练时间，并结合DTW算法，实现了手功能康复情况的自动评估。

关键词: 动态手势识别, 3D卷积神经网络, MediaPipe框架, 动态时间规整, 手功能康复

Abstract: To enable automatic and accurate quantitative evaluation of hand function rehabilitation in stroke patients, this paper proposes a skeleton-based gesture recognition and evaluation method. First, the MediaPipe framework is used to extract hand keypoints and connect them to form a hand skeleton, converting traditional RGB video datasets into hand skeleton datasets. Then, a 3D convolutional neural network (C3D) model is employed to train on and recognize hand functional movements. Based on correct recognition, further evaluation is conducted using the dynamic time warping (DTW) algorithm. The DTW distance between the motions of the healthy and affected hands performing the same action is calculated, aligning both temporally and spatially to represent the similarity in action execution. Experiments establish optimal DTW thresholds for distinguishing different rehabilitation ratings for each action, which serve as the criteria for quantitative evaluation.Results show that using skeletal data instead of traditional video improves gesture recognition accuracy to 99.01% and reduces training time. With the DTW algorithm, automatic hand function rehabilitation assessment is achieved.

Key words: dynamic gesture recognition, 3D convolutional neural network (CNN), MediaPipe framework, dynamic time warping (DTW), hand function rehabilitation

中图分类号: