Recording, annotation and Playback of in-house multimedia therapy exercises with Kinect, Leap and Myo


·  Patients needing physiotherapy are increasing at a very high rate as compared to the increase in the number of therapists.

·  Individual home based care is expensive and time consuming.

·  Most of the people cannot perform exercise correctly without help from a professional therapist.

·  A therapist has no way of knowing if a patient performed exercises at home correctly or not.

·  There is no way for a therapist to correct a person performing exercise at home.


·  We have developed a multimedia playing, recording and annotation environment that allows a therapist to record sample exercise for the patient to view and follow at home.

·  The patient can record her own exercise session while following the therapist recorded session.

·  The therapist can annotate the patient’s session to provide audio/visual feedback on top of the patient recorded exercise session.


·  Therapists can remotely monitor patients.

·  Patients can get audio/visual feedback from the system and correct their exercise postures.

·  More patients can be handled by therapists since they do not have to spend time travelling to the patients’ homes and can also give feedback when patient is not available.

·  Patients do not have to get appointments or wait in long queues to see the doctor.

·  The Big Data backend supports simultaneous recording and storage of very large number of patients simultaneously.

Patents Related to This Demo

1. A Synchronized Multimedia Engine for Therapy Recording, Playback, Annotation and Query in Big Data Environment

2. Physical Therapy with Mixed Reality

3. BCI (Brain Computer Interface) based VR (Virtual Reality) Therapy System for Paralyzed patients to recover muscle functions

4. Therapist-Assistant: Virtual Reality based Live Augmented Skeletal Therapy Management System

5. A serious game generator for disability persons based on patient disability level and/or therapist recommendation using non-invasive motion sensors

6. Gestpass: a multimodal gesture-based login system for providing assisted living services to physically challenged individuals

7. Mapping virtual 3d human model with the joint-motions of a disabled: a web-based 3d human model with natural user interface-based interaction capability for designing physical therapy

8. Immersive telepresence as occupational therapy for hemiplegic patients through VR technology

9. Hajjgeocaching - a real-life massive spatio-temporal online game for pilgrims with Motor Disability·

10. An electromyography (emg) signal and augmented reality based occupational hand therapy framework for people with hemiplegia

11. A gesture controlled mobile based camera-less smart home management solution for providing assisted living services to physically challenged individuals with low mobility

12. A non-invasive healthgis multimedia serious game therapy environment for providing ranges for motion of full-body joints to hemiplegic patient through forward and inverse kinematics

13. System and method for detecting and visualizing live kinetic and kinematic data for the musculoskeletal system

14. A web-based non-invasive way of detecting, tracking, recognizing and 3d visualizing live kinematic hand therapy data of hemiplegic patient