Kinect-enabled home-based rehabilitation system using Dynamic Time Warping and fuzzy logic
Graphical abstract
Introduction
Many countries are experiencing a rapid aging of their populations, are faced with the need to replace increasingly scarce skilled workers with technology-based substitutes. Part of this trend is the increased use of assistive technology to provide traditionally hospital-based medical services at home.
In the case of rehabilitation, traditional therapy is generally conducted in a hospital setting and requires direct supervision by a trained caregiver. The aim of home-based rehabilitation is to provide an in-home alternative to in-hospital rehabilitation. Home-based rehabilitation allows for great flexibility, allowing patients to tailor their rehabilitation program to individual preferences and schedules. In many countries today, home-based rehabilitation services enjoy government support, and the number of patients applying for such services has increased significantly in recent years [1]. Demand for medical services will soon outstrip the supply of registered physicians, necessitating the development of a more economically-viable home-based treatment program without the presence of a health care worker.
Such home-based solutions would also reduce the inconvenience of traveling to distant clinics for regular therapy services. Patients may not have enough time to go to the clinic or lack transportation. For many patients, frequent travel to the clinic is a significant economic burden. Moreover, the rehabilitation exercises conducted in a formal clinic setting may be insufficient to ensure the patient's recovery without supplementary rehabilitation exercise at home. To address these issues, physicians frequently prescribe a personal home-based rehabilitation exercise for patients, thus increasing the rehabilitation treatment effect and reducing the frequency of clinic visits. However, without professional supervision, home-based rehabilitation exercises could be, at best, ineffective and, at worst, unsafe.
In this paper, we describe our development of a Kinect-based system – the Kinect-enabled system for ensuring home-based rehabilitation (KEHR) using a Dynamic Time Warping (DTW) algorithm and fuzzy logic to ensure the effectiveness and safety of home-based rehabilitation. Using KEHR, the patient first performs a prescribed exercise in the presence of a health care professional. The exercise is recorded as a base for evaluating the patient's rehabilitation exercise at home, and these evaluations can be used as a reference for the patient to validate his/her exercise performance and to prevent adverse events. A summary report of the outcomes may also be automatically uploaded to a cloud setting for physicians to monitor the patient's progress and adjust the prescription.
The Kinect was launched on November 2010 as a webcam-style add-on peripheral for the Xbox 360 gaming console, enabling users to interact and control games without using through a natural user interface using gestures, voice or images rather than a controller or body sensors [2], [3].
Since its launch, software developers began to use the Kinect for other applications, raising the possibility of using it as part of a rehabilitation tool. Currently, the Kinect is being used in conjunction with PCs in ways that its designers could not have foreseen, from helping children with autism to helping doctors in the operating room [4].
The Kinect SDK for Windows provides detailed location, position and orientation information for up to two players standing in front of the Kinect sensor array. Previous devices had difficulty tracking human motion using a camera without body sensors. The process of extracting the human figure from video images proceeds in two phases: (1) preprocessing that detects the human object silhouette and extracts silhouette descriptors and (2) pose estimation that quantitatively characterizes and localizes human limbs in each frame. The Kinect not only provides skeletal tracking capabilities without body sensors but also furnishes a low-cost mechanism for developing home-based systems to improve our daily life.
Euclidean distance metrics are commonly used for comparing two time series due to their ease of calculation. In this research, however, patients may perform their recorded exercises for different durations and under varying conditions. Euclidean distance measurement between frames is generally unsuitable for comparing two sequential movements of different durations because it is sensitive to even a small distortion in time axis. A distance measurement called Dynamic Time Warping (DTW) has been widely applied in speech processing [5], [6], [7] and can be used to address the issue of time axis distortion.
The nonlinear dynamic time-warped alignment allows for the calculation of a more intuitive distance measurement. A method which does not suffer from the abovementioned shortcoming of Euclidean distance is needed to determine the similarity between the standard and the patient's actual exercise performance.
DTW algorithm is an extremely efficient time-series similarity measurement which minimizes the effects of shifting and distortion in time by allowing the elastic transformation of time series to detect similar shapes. These advantages are used here to compare two sequences of different durations to determine the similarity between the standard and the patient's exercise performance.
Physicians evaluate the trajectory and speed of rehabilitation exercise mainly based on their experience and subjective criteria without using more precise and measurable computer-based values. Therefore, we cannot set a value of trajectory and speed to evaluate results using traditional logic theory. In contrast with traditional logic theory, where traditional binary sets have a two-valued logic (i.e., true or false), fuzzy logic variables may have a truth value that ranges between 0 and 1.
The proposed approach collects the physician's subjective evaluation and uses the DTW algorithm to collect trajectory and speed data to build a fuzzy inference model of the physician's subjective evaluation.
Section snippets
Device-assisted rehabilitation
Many studies have used industrial motion sensors [7] and the Nintendo Wii Remote [8] to assist physicians and patients. Video games or virtual reality (VR) environments have been used to divert the patient's attention during the rehabilitation process, thus relieving feelings of discomfort and boredom, and physicians have used the rehabilitation data collected by the devices to better understand and monitor the patient's rehabilitation process. These studies have shown that motion sensors are
Requirement study and data preprocessing
Exercise is a basic component of most rehabilitation programs. In home-based rehabilitation, a physician thoroughly assesses the patient's condition and limitations before prescribing tailored, low-impact exercises which would not aggravate the injury for the patient to practice at home. The physician demonstrates the exercises and these demonstrations are recorded on video for the patient to follow at home. In this paper, we describe the development of a Kinect-based system (KEHR) to ensure
Hardware and software configurations
The KEHR system was implemented on Windows 7, using the Kinect for Windows SDK version 1.5, and the required code was compiled in Visual Studio 2010. The application was built on the Windows Presentation Foundation (WPF) [36]. Expression Blend 4 (EB4) SP1 [37] was used to design the user interface. Matlab R2011b was used to build the fuzzy inference. Overall hardware and software configurations are summarized in Table 11.
Scenario 1: patient's/user's perspective
Joe is a businessman suffering from shoulder pain. At the clinic, the
System usability evaluations
In this research, performance and usability/readability of the proposed KEHR system are of utmost importance. System performance was evaluated by against that provided by a physician by use of a questionnaire. In-hospital KEHR exercise sessions were recorded with four different users, each in the presence of a physician, while 30 completed the questionnaire, with results illustrated in Table 12.
We tested KEHR on three different shoulder rehabilitation exercises (Fig. 21) performed by four
Conclusion
KEHR could make a crucial contribution to future home-based rehabilitation. From the physician's point of view, in-hospital rehabilitation is sometimes insufficient for the patient's recovery and must be supplemented by consistent rehabilitation exercise at home. In addition, KEHR can ensure patients accurate reproduce the required rehabilitation exercises. From the patient's point of view, KEHR can at least partially resolve the inconvenience in traveling to the clinic for regular therapy
References (37)
- et al.
Home-based cardiac rehabilitation versus hospital-based rehabilitation: a cost effectiveness analysis
Int. J. Cardiol.
(2007) - et al.
A limb action detector enabling people with multiple disabilities to control environmental stimulation through limb action with a Nintendo Wii Remote Controller
Res. Dev. Disabil.
(2010) - et al.
A gesture recognition system to transition autonomously through vocational tasks for individuals with cognitive impairments
Res. Dev. Disabil.
(2011) - et al.
A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities
Res. Dev. Disabil.
(2011) - et al.
Early recognition of upper limb motor tasks through accelerometers: real-time implementation of a DTW-based algorithm
Comput. Biol. Med.
(2011) Fuzzy sets
Inform. Control
(1965)Is there a need for fuzzy logic?
Inform. Sci.
(2008)- et al.
A fuzzy logic controller for a dynamic plant
Int. J. Man-Machine Stud.
(1975) - et al.
Evaluation of student performance in laboratory applications using fuzzy logic
Procedia Soc. Behav. Sci.
(2010) - et al.
An interactive computer graphics interface for the introduction of fuzzy inference in environmental education
Interact. Comput.
(2006)
E3: Microsoft shows off Gesture Control Technology for Xbox 360
E3 2009: Microsoft at E3 Several Metric Tons of Press Releaseapalloza
Kinect Effect Magic pushes beyond the Living Room
Performance tradeoffs in dynamic time warping algorithms for isolated word recognition
IEEE Trans. Acoustics Speech Signal Process.
Dynamic programming algorithm optimization for spoken word recognition
IEEE Trans. Acoustics Speech Signal Process.
Low-cost accelerometry-based posture monitoring system for stroke survivors
Kinect Games
Real-time classification of dance gestures from skeleton animation
Cited by (123)
Artificial Intelligence for skeleton-based physical rehabilitation action evaluation: A systematic review
2023, Computers in Biology and MedicineAutomatic Evaluation of Motor Rehabilitation Exercises Based on Deep Mixture Density Neural Networks
2022, Journal of Biomedical InformaticsPostural control assessment via Microsoft Azure Kinect DK: An evaluation study
2021, Computer Methods and Programs in BiomedicineComputerized Assessment of Motor Imitation as a Scalable Method for Distinguishing Children With Autism
2021, Biological Psychiatry: Cognitive Neuroscience and Neuroimaging