System and method for aiding communication

Patent metadata

Field	Value
Publication number	US-11086473-B2
Application number	US-201716321286-A
Country	US
Kind code	B2
Filing date	May 23, 2017
Priority date	Jul 28, 2016
Publication date	Aug 10, 2021
Grant date	Aug 10, 2021

Abstract

System and method for aiding communication for subjects suffering from paralysis of muscles controlled by peripheral nervous system are disclosed. A method for aiding communication for said subjects includes capturing, from a plurality of sensors, sensor data generated based on an interaction of a subject with an interactive UI. The plurality of sensors includes one or more body-parts movement tracking sensors for tracking motion of body-parts of the subject and one or more physiological signal sensors for monitoring physiological signals generated from the subject during the interaction. A plurality of model parameters indicative of characteristics of the subject related to the interaction are determined based on the sensor data. The navigation at the interactive UI is controlled on the plurality of model parameters.

First claim

The invention claimed is:

1. A processor-implemented method for aiding communication, the method comprising:
capturing, from a plurality of sensors, sensor data generated based on an interaction of a subject with an interactive UI, via one or more hardware processors, the plurality of sensors comprising one or more body-parts movement tracking sensors for tracking motion of body-parts of the subject and one or more physiological signal sensors for monitoring physiological signals generated from the subject during the interaction;
determining, based on the sensor data, a plurality of model parameters indicative of characteristics of the subject related to the interaction, via the one or more hardware processors, wherein the sensor data is collective sensor data from the one or more body-parts movement tracking sensors and the one or more physiological signal sensors, wherein the plurality of model parameters are determined by training a model, and wherein training the model comprises:
monitoring dynamically, the sensor data generated based on the interaction of the subject with the interactive UI to determine a consistent change in the interaction, wherein determining the consistent change in the interaction comprises verifying, for a threshold number of times, change in the interaction of the subject with the interactive UI and a changed response of the subject due to progression of medical condition of the subject, to validate whether the changed response is consistent during the threshold number of times, and
automatically updating, on determination of the consistent change, one or more model parameters from the plurality of model parameters based on the sensor data, and

controlling navigation at the interactive UI based on the plurality of model parameters, via the one or more hardware processors.

2. The method as claimed in claim 1 , wherein the one or more body-parts movement tracking sensors comprises Inertial Motion Unit (IMU), an electroencephalogram (EEG) sensor, eye tracking sensor, gyroscope, and magnetometer.

3. The method as claimed in claim 1 , wherein the one or more physiological signal sensors comprises Galvanic Skin Response sensor and Photoplethysmogram sensor.

4. The method as claimed in claim 1 , further comprising training the model for determining the plurality of model parameters, wherein training the model in an offline mode comprises:
performing a plurality of known tasks and capturing a training sensor data during said performing, wherein the plurality of known tasks comprises interacting with the interactive UI; and
obtaining a plurality of training model parameters based on the training sensor data, the plurality of training model parameters facilitates in training the model.

5. The method as claimed in claim 1 , wherein controlling the navigation at the interactive UI comprises one or more of: indicating highlighting at least one element on the interactive UI, generating an audio output at the interactive UI and generating a visual output at the interactive UI.

6. The method as claimed in claim 1 , further comprising adapting the interactive UI based on the updating of the one or more model parameters.

7. The method as claimed in claim 1 , wherein the interactive UI comprises an on-screen keyboard layout.

8. The method as claimed in claim 1 , further comprising storing the updated one or more model parameters at a server to derive an adapted interactive UI.

9. The method as claimed in claim 8 , wherein adapting the interactive UI comprises adapting one or more elements of the interactive UI.

10. The method as claimed in claim 8 , wherein adapting the interactive UI comprises adapting an orientation of a layout of the interactive UI.

11. A system for aiding communication, the system comprising:
one or more memories storing instructions; and
one or more hardware processors coupled to the one or more memories, wherein said one or more hardware processors are configured by said instructions to:
capture, from a plurality of sensors, sensor data generated based on an interaction of a subject with an interactive UI, the plurality of sensors comprising one or more body-parts movement tracking sensors for tracking motion of body-parts of the subject and one or more physiological signal sensors for monitoring physiological signals generated from the subject during the interaction;
determine, based on the sensor data, a plurality of model parameters indicative of characteristics of the subject related to the interaction, wherein the sensor data is collective sensor data from the one or more body-parts movement tracking sensors and the one or more physiological signal sensors, wherein the plurality of model parameters are determined by training a model, and wherein training the model comprises:
monitor dynamically, the sensor data generated based on the interaction of the subject with the interactive UI to determine a consistent change in the interaction, wherein determining the consistent change in the interaction comprises verifying, for a threshold number of times, change in the interaction of the subject with the interactive UI and a changed response of the subject due to progression of medical condition of the subject, to validate whether the changed response is consistent during the threshold number of times, and
automatically update, on determination of the consistent change, one or more model parameters from the plurality of model parameters based on the sensor data, and

control navigation at the interactive UI based on the plurality of model parameters.

12. The system as claimed in claim 11 , wherein the one or more body-parts movement tracking sensors comprises Inertial Motion Unit (IMU), an electroencephalogram (EEG) sensor, eye tracking sensor, gyroscope, and magnetometer.

13. The system as claimed in claim 11 , wherein the one or more physiological signal sensors comprises Galvanic Skin Response sensor and Photoplethysmogram sensor.

14. The system as claimed in claim 11 , wherein the one or more hardware processors are further configured by the instructions to train the model for determining the plurality of model parameters, and wherein to train the model in an offline mode, the one or more hardware processors are further configured by the instructions to:
perform a plurality of known tasks and capturing a training sensor data during said performing, wherein the plurality of known tasks comprises interacting with the interactive UI; and
obtain a plurality of training model parameters based on the training sensor data, the plurality of training model parameters facilitates in training the model.

15. The system as claimed in claim 11 , wherein to control the navigation at the interactive UI, the one or more hardware processors are further configured by the instructions to perform one or more of: indicate highlighting at least one element on the interactive UI, generate an audio output at the interactive UI and generate a visual output at the interactive UI.

16. The system as claimed in claim 11 , wherein the one or more hardware processors are further configured by the instructions to:
adapt the interactive UI based on the updating of the one or more model parameters; and
store the updated one or more model parameters at a server to derive the adapted interactive UI.

17. The system as claimed in claim 16 , wherein to adapt the interactive UI, the one or more hardware processors are further configured by the instructions to adapt one or more elements of the interactive UI.

18. The system as claimed in claim 16 , wh

Assignees

• Tata Consultancy Services Ltd

Inventors

• Vishwakarma Arpit
• Sinha Aniruddha
• Manna Ratnamala
• Chatterjee Debatri
• Vedraj
• Chakravarty Kingshuk
• Das Rajat Kumar
• Mehrotra Anagha Nikhil
• Pal Arpan
• Gavas Rahul Dasharath
• Khasnobish Anwesha
• Das Abhijit

Classifications

• A61B5/1114 Primary

  Tracking parts of the body · CPC title

• G06N20/00

  Machine learning · CPC title

• A61F4/00

  Methods or devices enabling patients or disabled persons to operate an apparatus or a device not forming part of the body · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• H04N7/185

  from a mobile camera, e.g. for remote control · CPC title