Elsevier

Applied Soft Computing

Volume 22, September 2014, Pages 652-666
Applied Soft Computing

Kinect-enabled home-based rehabilitation system using Dynamic Time Warping and fuzzy logic

https://doi.org/10.1016/j.asoc.2014.04.020Get rights and content

Highlights

  • The exercise performed in conventional rehabilitation contexts may be insufficient to ensure the patient's speedy recovery.

  • A KEHR system to promote effective home-based rehabilitation without the immediate supervision of a physician is proposed.

  • The proposed KEHR resolve the inconvenience in traveling to the clinic for regular therapy services.

Abstract

Most formal rehabilitation facilities are situated in a hospital or care center setting, which may not always be conveniently accessible for patients, especially those in geographically isolated areas. Home-based rehabilitation has potential to offer greater accessibility and thus increase consistent uptake. In addition, the exercise performed in conventional rehabilitation contexts may be insufficient to ensure the patient's speedy recovery, with complimentary rehabilitation exercises at home required to make a difference. The goal is to provide effective home-based rehabilitation offering outcomes similar to those obtained through hospital-based rehabilitation under the supervision of an occupational therapist. This paper presents the development of a Kinect-based system for ensuring home-based rehabilitation using a Dynamic Time Warping (DTW) algorithm and fuzzy logic. The ultimate goal is to assist patients in conducting safe and effective home-based rehabilitation without the immediate supervision of a physician.

Introduction

Many countries are experiencing a rapid aging of their populations, are faced with the need to replace increasingly scarce skilled workers with technology-based substitutes. Part of this trend is the increased use of assistive technology to provide traditionally hospital-based medical services at home.

In the case of rehabilitation, traditional therapy is generally conducted in a hospital setting and requires direct supervision by a trained caregiver. The aim of home-based rehabilitation is to provide an in-home alternative to in-hospital rehabilitation. Home-based rehabilitation allows for great flexibility, allowing patients to tailor their rehabilitation program to individual preferences and schedules. In many countries today, home-based rehabilitation services enjoy government support, and the number of patients applying for such services has increased significantly in recent years [1]. Demand for medical services will soon outstrip the supply of registered physicians, necessitating the development of a more economically-viable home-based treatment program without the presence of a health care worker.

Such home-based solutions would also reduce the inconvenience of traveling to distant clinics for regular therapy services. Patients may not have enough time to go to the clinic or lack transportation. For many patients, frequent travel to the clinic is a significant economic burden. Moreover, the rehabilitation exercises conducted in a formal clinic setting may be insufficient to ensure the patient's recovery without supplementary rehabilitation exercise at home. To address these issues, physicians frequently prescribe a personal home-based rehabilitation exercise for patients, thus increasing the rehabilitation treatment effect and reducing the frequency of clinic visits. However, without professional supervision, home-based rehabilitation exercises could be, at best, ineffective and, at worst, unsafe.

In this paper, we describe our development of a Kinect-based system – the Kinect-enabled system for ensuring home-based rehabilitation (KEHR) using a Dynamic Time Warping (DTW) algorithm and fuzzy logic to ensure the effectiveness and safety of home-based rehabilitation. Using KEHR, the patient first performs a prescribed exercise in the presence of a health care professional. The exercise is recorded as a base for evaluating the patient's rehabilitation exercise at home, and these evaluations can be used as a reference for the patient to validate his/her exercise performance and to prevent adverse events. A summary report of the outcomes may also be automatically uploaded to a cloud setting for physicians to monitor the patient's progress and adjust the prescription.

The Kinect was launched on November 2010 as a webcam-style add-on peripheral for the Xbox 360 gaming console, enabling users to interact and control games without using through a natural user interface using gestures, voice or images rather than a controller or body sensors [2], [3].

Since its launch, software developers began to use the Kinect for other applications, raising the possibility of using it as part of a rehabilitation tool. Currently, the Kinect is being used in conjunction with PCs in ways that its designers could not have foreseen, from helping children with autism to helping doctors in the operating room [4].

The Kinect SDK for Windows provides detailed location, position and orientation information for up to two players standing in front of the Kinect sensor array. Previous devices had difficulty tracking human motion using a camera without body sensors. The process of extracting the human figure from video images proceeds in two phases: (1) preprocessing that detects the human object silhouette and extracts silhouette descriptors and (2) pose estimation that quantitatively characterizes and localizes human limbs in each frame. The Kinect not only provides skeletal tracking capabilities without body sensors but also furnishes a low-cost mechanism for developing home-based systems to improve our daily life.

Euclidean distance metrics are commonly used for comparing two time series due to their ease of calculation. In this research, however, patients may perform their recorded exercises for different durations and under varying conditions. Euclidean distance measurement between frames is generally unsuitable for comparing two sequential movements of different durations because it is sensitive to even a small distortion in time axis. A distance measurement called Dynamic Time Warping (DTW) has been widely applied in speech processing [5], [6], [7] and can be used to address the issue of time axis distortion.

The nonlinear dynamic time-warped alignment allows for the calculation of a more intuitive distance measurement. A method which does not suffer from the abovementioned shortcoming of Euclidean distance is needed to determine the similarity between the standard and the patient's actual exercise performance.

DTW algorithm is an extremely efficient time-series similarity measurement which minimizes the effects of shifting and distortion in time by allowing the elastic transformation of time series to detect similar shapes. These advantages are used here to compare two sequences of different durations to determine the similarity between the standard and the patient's exercise performance.

Physicians evaluate the trajectory and speed of rehabilitation exercise mainly based on their experience and subjective criteria without using more precise and measurable computer-based values. Therefore, we cannot set a value of trajectory and speed to evaluate results using traditional logic theory. In contrast with traditional logic theory, where traditional binary sets have a two-valued logic (i.e., true or false), fuzzy logic variables may have a truth value that ranges between 0 and 1.

The proposed approach collects the physician's subjective evaluation and uses the DTW algorithm to collect trajectory and speed data to build a fuzzy inference model of the physician's subjective evaluation.

Section snippets

Device-assisted rehabilitation

Many studies have used industrial motion sensors [7] and the Nintendo Wii Remote [8] to assist physicians and patients. Video games or virtual reality (VR) environments have been used to divert the patient's attention during the rehabilitation process, thus relieving feelings of discomfort and boredom, and physicians have used the rehabilitation data collected by the devices to better understand and monitor the patient's rehabilitation process. These studies have shown that motion sensors are

Requirement study and data preprocessing

Exercise is a basic component of most rehabilitation programs. In home-based rehabilitation, a physician thoroughly assesses the patient's condition and limitations before prescribing tailored, low-impact exercises which would not aggravate the injury for the patient to practice at home. The physician demonstrates the exercises and these demonstrations are recorded on video for the patient to follow at home. In this paper, we describe the development of a Kinect-based system (KEHR) to ensure

Hardware and software configurations

The KEHR system was implemented on Windows 7, using the Kinect for Windows SDK version 1.5, and the required code was compiled in Visual Studio 2010. The application was built on the Windows Presentation Foundation (WPF) [36]. Expression Blend 4 (EB4) SP1 [37] was used to design the user interface. Matlab R2011b was used to build the fuzzy inference. Overall hardware and software configurations are summarized in Table 11.

Scenario 1: patient's/user's perspective

Joe is a businessman suffering from shoulder pain. At the clinic, the

System usability evaluations

In this research, performance and usability/readability of the proposed KEHR system are of utmost importance. System performance was evaluated by against that provided by a physician by use of a questionnaire. In-hospital KEHR exercise sessions were recorded with four different users, each in the presence of a physician, while 30 completed the questionnaire, with results illustrated in Table 12.

We tested KEHR on three different shoulder rehabilitation exercises (Fig. 21) performed by four

Conclusion

KEHR could make a crucial contribution to future home-based rehabilitation. From the physician's point of view, in-hospital rehabilitation is sometimes insufficient for the patient's recovery and must be supplemented by consistent rehabilitation exercise at home. In addition, KEHR can ensure patients accurate reproduce the required rehabilitation exercises. From the patient's point of view, KEHR can at least partially resolve the inconvenience in traveling to the clinic for regular therapy

References (37)

  • A. Pham

    E3: Microsoft shows off Gesture Control Technology for Xbox 360

    (2009)
  • Microsoft

    E3 2009: Microsoft at E3 Several Metric Tons of Press Releaseapalloza

    (2009)
  • Microsoft Research

    Kinect Effect Magic pushes beyond the Living Room

    (2011)
  • C. Myers et al.

    Performance tradeoffs in dynamic time warping algorithms for isolated word recognition

    IEEE Trans. Acoustics Speech Signal Process.

    (1980)
  • H. Sakoe et al.

    Dynamic programming algorithm optimization for spoken word recognition

    IEEE Trans. Acoustics Speech Signal Process.

    (1978)
  • S. Arteaga et al.

    Low-cost accelerometry-based posture monitoring system for stroke survivors

  • Microsoft

    Kinect Games

    (2012)
  • M. Raptis et al.

    Real-time classification of dance gestures from skeleton animation

  • Cited by (123)

    View all citing articles on Scopus
    View full text