Therapy Authoring to Create Customized and Personalized Therapies

Problem:
·  Physiotherapy exercises are boring.
·  Forcing young disabled children to perform exercises can be specially challenging.
·  Two thirds of the patients do not perform exercises correctly at home.
·  A therapist has no way of knowing if a patient performed exercises at home correctly or not.
·  There is no way for a therapist to correct a person performing exercise at home.
·  Recovery in many patients is slow and cannot be measured accurately by instruments currently in use.

The Solution:
·  We have developed an online flying game that can be played by a patient using hand and body gestures.
·  The gestures used to play the game are the ones required to perform physical therapy.
·  The therapist can customize the game with required gestures and difficulty level, depending on the current state of the patient.
·  Off the shelf multi-sensory environment will allow patients to play the games while the kinematic data will be available to the doctor for further analysis and deciding quality of improvement.

The Advantages:
·  Immersive and engaging game play removes boredom from exercise.
·  Difficulty level of the game can be changed automatically by the system or remotely by the therapist according to the recovery behavior of each patient.
·  Visual feedback allows the patient to correct the exercise immediately. Wrong movements do not score points.
·  Powerful gesture detection sensors record minute differences in joint movements and report full details of the exercise performed at home to the therapist.
·  Advanced analytics can provide detailed reports, graphs and charts regarding recovery.
·  Social media integration can help people recover quickly by sharing their experiences and supporting each other.
·  Big Data back-end can store data generated by millions of users simultaneously.
·  Computational Intelligence can be used to study and discover new recovery parameters and patterns based on Big Data analysis.

Multi-Sensory Gesture Based Login for People with Special needs and Doctors in Clinical Environment.

Problem

• Disabled people are not always able to use the keyboard properly.
• Entering passwords is a special problem. Slow entry can allow others to see the password.
• Doctors in clinical environment wants hands-free access to computers in cases where hands are engaged.

Solution

• We have developed a multi-sensor-based password entry environment for the physically challenged that allows users to log in using hand gestures.
• The gestures can be detected even in a dark room using only an infra-red light or through reading the EMG signals produced by muscle activity due to movement of hand.

Advantages

• A disabled person can enter password at her own speed without having to worry about privacy issues.
• For alphanumeric characters, fingers from both hands as well as arm position can be used to define all characters.
• For numeric only passwords, a combination of simple and easy to remember gestures can be used.
• EMG signal-based solution can be used at ATMs because passwords can be entered with the hand in pocket and password entry cannot be recognized with a hidden video camera.

·  Doctors often cannot touch keyboards while performing medical procedures due to fear of contamination. They can use this facility to log on to systems without touching the keyboard.

Gesture based Serious Games for Physical Ther

Problem

• Disabled people are not always able to use the keyboard properly.
• Entering passwords is a special problem. Slow entry can allow others to see the password.
• Doctors in clinical environment wants hands-free access to computers in cases where hands are engaged.

Solution

• We have developed a multi-sensor-based password entry environment for the physically challenged that allows users to log in using hand gestures.
• The gestures can be detected even in a dark room using only an infra-red light or through reading the EMG signals produced by muscle activity due to movement of hand.

Advantages

• A disabled person can enter password at her own speed without having to worry about privacy issues.
• For alphanumeric characters, fingers from both hands as well as arm position can be used to define all characters.
• For numeric only passwords, a combination of simple and easy to remember gestures can be used.
• EMG signal-based solution can be used at ATMs because passwords can be entered with the hand in pocket and password entry cannot be recognized with a hidden video camera.

·  Doctors often cannot touch keyboards while performing medical procedures due to fear of contamination. They can use this facility to log on to systems without touching the keyboard.

Gesture based Clinical Data Browsing for Patients and Medical Doctors

The Problem
·  Many disabled people cannot use keyboards and mouse.
·  A lot of solutions exist but they all are some kind of touch devices that need specific amount of pressure to be applied to a surface.
·  These solutions cannot be configured for every level of disability.
·  These solutions work for some people while they do not work for others, depending on each person’s type of disability.
·  Touch screens are also not convenient for many disabled people.
·  Gestures are used for occupational therapeutic movements

Solution
·  We have developed a solution to browse the internet using gestures.
·  We use smart sensors that detect gestures performed by the user with her hands in the air and convert them to mouse and keyboard clicks and events.
·  We can use different kind of devices to detect different kind of gestures and use them for browsing.
·  We can configure the gestures according to the type of disability of each user. Each user can select the gestures she feels most comfortable with and use them for the required tasks.
·  We can also configure the sensitivity of each gesture through software according to the level of disability of each patient.

Advantage
·  A patient can browse the internet easily without additional help.
·  We can configure the gestures according to the type of disability of each user. Each user can select the gestures she feels most comfortable with and use them for the required tasks.
·  We can also configure the sensitivity of each gesture through software according to the level of disability of each patient.
·  Doctors wearing gloves who cannot use a touch screen can use this technology to browse material on the screen, zoom in/out images etc.

Multimedia Big Data support of e-Health applications

Problem
·  Health sensors produce a large amount of data.
·  With the increase in use of health sensors by patients, the amount of data produced is increasing at an alarming speed. This data needs to be stored immediately without delay.
·  Advanced analytical computations on stored data are needed to extract useful information regarding a patient’s health.
·  Finding recovery patterns across different sectors of the society require very high volume data analysis.

Solution
·  We have developed a Big Data based multimedia platform to support e-health Big Data.
·  It allows high speed data collection, high volume data storage and high performance data analytics using a Hadoop based customized Big Data solution.

Advantages
•  Sensor based e-health technologies can be used with ease.
•  Advanced data analytics can help study and discover new recovery metrics not imagined before.
•  E-health data stored in the cloud can be easily shared between remote locations.

Value to the Kingdom
•  A national e-Health Big Data infrastructure can be developed around this solution to provide data storage facility for all patients in the kingdom.
•  All hospitals in the kingdom can be linked to the high performance data center to support knowledge sharing and collaboration.
•  With the implementation of a standard Electronic Health Record (EHR) solution, patients in the kingdom will be able to move between hospitals easily by electronically sharing their medical data between hospitals and physicians.

Recording, annotation and Playback of in-house multimedia therapy exercises with Kinect, Leap and Myo

Problem

• Patients needing physiotherapy are increasing at a very high rate as compared to the increase in the number of therapists.
• Individual home based care is expensive and time consuming.
• Most of the people cannot perform exercise correctly without help from a professional therapist.
• A therapist has no way of knowing if a patient performed exercises at home correctly or not.
• There is no way for a therapist to correct a person performing exercise at home.

Solution

• We have developed a multimedia playing, recording and annotation environment that allows a therapist to record sample exercise for the patient to view and follow at home.
• The patient can record her own exercise session while following the therapist recorded session.
• The therapist can annotate the patient’s session to provide audio/visual feedback on top of the patient recorded exercise session.

Advantages

• Therapists can remotely monitor patients.
• Patients can get audio/visual feedback from the system and correct their exercise postures.
• More patients can be handled by therapists since they do not have to spend time travelling to the patients’ homes and can also give feedback when patient is not available.
• Patients do not have to get appointments or wait in long queues to see the doctor.

·  The Big Data backend supports simultaneous recording and storage of very large number of patients simultaneously.

Research on Pilgrim e-Health Services

Problem
·  Pilgrims from more than 140 countries come every year to Makkah for Hajj and Umrah.
·  Most of them are illiterate or do not understand enough Arabic language to explain their medical problems.
·  They do not know the locality and hence may not know how to find a hospital in case of emergency or whom to report any medical conditions.
·  Currently, pilgrims have no way of informing people of interest such as hajj agent, medical doctor, family members, fellow pilgrims etc. at nearby locations about the health issues, location of the patient, vital information of the patient etc.

Solution
·  We have developed a Hajj and Umrah companion app and family connect app that provide health related services to pilgrims and connect pilgrims with their family members, co-pilgrims, hajj agencies, medical facilities and emergency department.
·  A pilgrim can find the nearest hospitals and medical facilities through the developed apps.
·  In case of emergency, the pilgrim can find the phone number of ambulance immediately and call directly from the phone or use the free SMS service to share vital as well as location information.
·  The traffic feature in the app can help a person reach the nearest medical facility in time by avoiding congested paths.
·  The translation feature can help the patient explain his problem to the medical personnel.
·  A health form is provided to the pilgrim to report any health-related vital information to the authorities.

Demo Scenario
The presenter will show the Health Form. Basic medical info can be sent to authorities through this form.
In places of interest, a pilgrim can see the nearest medical facility and get directions to get there.
The traffic update screen can tell him about possible accidents along the way so that he can change his route if needed.
In emergency, the pilgrim can find the phone number to call in case of medical emergency. In our new interface, the number will be callable directly from the screen.
In the same screen, the Free SMS can be used to send SMS in a medical emergency.
Through the news screen, a person can follow up with health-related news.
Through the translation option, the patient can explain his problem to the medical staff if he does not know their language.
In the complaints section, the pilgrim can report health related problems in the area. He can also take photographs and send them through the app.
For all kinds of health-related problems, the pilgrim can contact the Haj Agency immediately through the app.

Smart home appliance control using natural user interfaces: occupational therapy imperative

Problem

Many patients recovering from stroke need Occupational Therapy to perform daily living activities in a normal manner. However,

• Occupational Therapy requires external help.
• Many disabled people cannot move around the house to perform daily chores.

Challenge

• Can we develop a system that allows a disabled person to control appliances in the house without leaving her seat?
• Can we convert this daily life activity into an occupational therapy activity that can be monitored and recorded?
• Can we measure the progress of the patient?

Solution

• We have developed a smart home solution that allows a person to control home appliances such as light bulbs, fans and air conditioners remotely from her seat by making real gestures to control virtual knobs and switches.
• The activity performed by the patient is silently recorded by the system to measure recovery metrics.
• Our software platform runs on both PC and smartphone.

Advantages

• Smart home control allows a disabled person to manage home appliances without leaving her seat.
• The smartphone based solution is highly mobile and can be easily used from anywhere inside the house.
• The system supports a variety of sensors to detect gestures, such as Kinect2, Leap Motion and Myo.
• A large number of gestures can be detected by the system supporting different kinds of movements.
• A wide array of controllable hardware is supported by the software platform, such as Philips Hue and LIFX devices.
• Visual feedback allows the person to correct wrong gestures immediately.
• Encouraging interface artifacts are incorporated into the environment to make the OT experience enjoyable and to speed up the recovery process.
• Quantification of the recovery process allows gamification of the therapy environment, which increases the immersiveness of the experience, resulting in increased adherence of the patient to the proper therapy procedure.
• Integration with social media allows patients to create a collaborative environment for support and recovery.
• Recovery metrics are recorded seamlessly without intervention in the patient’s daily work routine.
• Therapists can monitor patients’ recovery metrics remotely.
• Big Data back end can track millions of patients simultaneously.

·  Big Data analytics can help researchers study and discover new recovery patterns and metrics.

Mapping Physical Human Joint-motion in Physical World with Virtual Reality 3D Skeleton Through Multi-Sensory Environment

Problem
· Two thirds of the patients do not perform exercises correctly at home.
· A patient needs constant coaching from the therapist while performing exercises.
· A therapist has no way of knowing if a patient performed exercises at home correctly or not.
· There is no way for a therapist to correct a person performing exercise at home.

Solution
· We have developed a sensor-based motion capture environment that allows a therapist to record a therapy.
· The therapist can make a video recording of the therapy and upload it to a cloud storage area for the patient to download and view while performing the exercise.
· The patient can see her avatar/stick figure superimposed onto the stick figure/avatar of the patient while performing exercise.

Advantages
· Powerful gesture detection sensors record minute differences in joint movements and report full details of the exercise performed at home to the therapist.
· Advanced analytics can provide detailed reports, graphs and charts regarding recovery.
· Visual feedback allows the patient to correct exercise immediately.
· Social media integration can help people recover quickly by sharing their experiences and supporting each other.
· Big Data back end can store data generated by millions of users simultaneously.
· Computational Intelligence can be used to study and discover new recovery parameters and patterns based on Big Data analysis.
· The 3D skeleton displayed on the screen allows the user to explain to the patient the exact details of her disability which may help her in focusing on the right group of muscles to perform the exercise properly.

Value to the Kingdom of Saudi Arabia:
· The 3D skeleton can be used for pedagogical purposes and different parts of the skeleton can be labeled in Arabic to make it easy to understand for both patients and students.

Multi-sensory, Non-Invasive Whole-Body Range of Motion Detection for Quality of Improvement Measurement

Problem
· Goniometers are manual devices for measuring joint Range of Motion (ROM) and hence are not very accurate.
· Minor recovery in ROM in patients cannot be measured using goniometers.
· Use of goniometers requires the help of a trained person.
· A patient needs to get an appointment at the clinic to measure her improvement with a goniometer.
· A patient seeing day-to-day improvement will be encouraged to do more exercise and recover quickly. This cannot be done with a goniometer due to the above mentioned reasons.

Solution
· We have developed a simple software application that can use devices like Kinect and Leap motion to measure joint ROM.
· The Kinect device can measure 25 joints in the human body 60 times per second while the Leap device can measure all hand joints at 30-200 times per second.

Advantages
• ROM can be detected easily anytime without the help of the therapist.
• Patients can know their improvement pace through detailed measurements.
• Gamification of the technology can lead to immersive environments for providing physiotherapy to remote patients.
• Motion detection facility provided by the sensors can be used to detect gestures which can in turn be used to perform different chores around the house using a computer and smart home appliances, making daily life activities easier for disabled people.
• The large amount of data collected through these devices can be used to study and discover new recovery metrics.
• The 3D skeleton displayed on the screen allows the user to explain to the patient the exact details of her disability which may help her in focusing on the right group of muscles to perform the exercise properly.
• The 3D skeleton can be used for pedagogical purposes.

Gesture based map browsing for physically challenged

The Problem
· Map browsing is a common activity, especially while on the road. Many disabled people cannot browse maps because their hands are busy while managing wheelchairs or other specially crafted driving contraptions.
· Interacting with maps while driving is dangerous for normal people also, even while using touch screens.
· Different people have different kinds of disabilities and hence most of the specialized hardware devised for people with a particular disability does not work for other people.
· Touch screens are also not convenient for many people.

The Solution
· We have developed a solution to browse maps using gestures.
· We use smart sensors that detect gestures performed by the user with her hands in the air and convert them to map browsing actions.
· We can use different kind of devices to detect different kind of gestures and use them for map browsing.
· We can also configure the sensitivity of each gesture through software according to the level of disability of each patient.

The Advantage
· A patient can browse maps easily without additional help.
· We can configure the gestures according to the type of disability of each user. Each user can select the gestures she feels most comfortable with and use them for the required tasks.
· We can also configure the sensitivity of each gesture through software according to the level of disability of each patient.
· The solution is helpful for both disabled and normal people.