System and method for predicting performance to control interventions by assistive technologies

Patent metadata

Field	Value
Publication number	US-11288977-B1
Application number	US-201816100921-A
Country	US
Kind code	B1
Filing date	Aug 10, 2018
Priority date	Oct 11, 2017
Publication date	Mar 29, 2022
Grant date	Mar 29, 2022

Abstract

In an embodiment of the present invention, a method for generating a prediction of ability of a subject to perform a task in a future time step includes receiving performance data corresponding to a performance of the subject on the task; receiving a plurality of biometric inputs computed based on physiological data during the performance of the subject on the task; identifying a numerical relationship between the performance data and the plurality of biometric inputs; generating a modulation parameter for each of the plurality of biometric inputs based on the identified numerical relationship; loading a plurality of state variable inputs produced by a generic model of performance; and generate the prediction of ability to perform the task at the prediction time, generated by a trained performance predictor based on biometric inputs predicted based on the modulation parameters.

First claim

What is claimed is:

1. A method comprising:
receiving, by a processor, performance data corresponding to a performance of a task comprising controlling a vehicle;
receiving, by the processor, a plurality of biometric inputs computed based on physiological data sensed by a first plurality of sensors during the performance of the task;
identifying, by the processor, a numerical relationship between the performance data and the plurality of biometric inputs;
generating, by the processor, a modulation parameter for each of the plurality of biometric inputs based on the identified numerical relationship;
loading, by the processor, a plurality of state variable inputs produced by a generic model of performance;
preparing, by the processor, a training set, wherein the training set includes performance data, the plurality of state variable inputs, the plurality of biometric inputs, and the modulation parameter for each of the plurality of biometric inputs;
training, by the processor, a machine learning model based on the training set to generate a trained performance predictor;
receiving, by the processor:
the trained performance predictor;
biometric inputs predicted by following the slope and intercept of each modulation parameter to a prediction time; and
cognitive model state variable predictions made by progressing the model forward to the prediction time;

generating, by the processor, a prediction of ability to perform the task at the prediction time, generated by the trained performance predictor based on the predicted biometric inputs;
determining, by the processor, that the ability to perform the task is below a threshold level; and
activating, by the processor, an automated system configured to directly control the vehicle to change movement in response to determining that the ability to perform the task is below the threshold level.

2. The method of claim 1 , wherein each of the plurality of state variable inputs is indicative of a skill level in the performance of the task.

3. The method of claim 1 , wherein the task is operating an aerospace vehicle.

4. The method of claim 1 , wherein the task is operating a semi-autonomous vehicle.

5. The method of claim 1 , wherein the task is performed on a training simulator.

6. The method of claim 1 , further comprising:
detecting, by the processor, that the prediction of ability to perform the task in the future time step is below a threshold; and
activating, by the processor, the automated system to assist in performing the task in response to determining that the prediction of ability to perform the task in the future time step is below the threshold.

7. The method of claim 1 , wherein the first plurality of sensors are configured to capture environmental data sensed in a region of where the task is performed and biometric data.

8. The method of claim 1 , wherein the performance data and the physiological data are captured within a plurality of time windows, each of the plurality of time windows corresponding to one of a plurality of trials involving the task.

9. The method of claim 8 , wherein, for each of the plurality of trials, a mean is calculated for the plurality of biometric inputs over the corresponding time window, the mean being used to identify the numerical relationship used in the training set, and used to identify the numerical relationship.

10. The method of claim 9 , wherein the corresponding time window over which the mean is calculated is different for each of the plurality of biometric inputs.

11. The method of claim 1 , wherein each of the plurality of biometric inputs corresponds to a time window of a plurality of time windows, the time window starting before a prompt for the task is presented.

12. The method of claim 1 , further comprising:
receiving, by a processor, a plurality of environmental sensor inputs from a second plurality of sensors, the plurality of environmental sensor inputs being captured during the performance of the task, wherein the modulation parameters encode the numerical relationship from the plurality of state variable inputs, the plurality of biometric inputs, and the plurality of environmental sensor inputs to the performance data.

13. A system for generating a prediction of a level of performance of a task in a future time step, the system comprising:
a computer system comprising of one or more processors and a memory;
a user input system in communication with the computer system; and
a plurality of sensors configured to measure physiological data, wherein the plurality of sensors are in communication with the computer system,
wherein the memory comprises a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of:
receiving, by a processor, performance data corresponding to the level of performance of a task comprising controlling a vehicle;
receiving, by the processor, a plurality of biometric inputs computed based on physiological data sensed by the plurality of sensors during the performance of the task;
identifying, by the processor, a numerical relationship between the performance data and the plurality of biometric inputs;
generating, by the processor, a modulation parameter for each of the plurality of biometric inputs based on the identified numerical relationship;
loading, by the processor, a plurality of state variable inputs produced by the performance data and a general model of performance;
preparing, by the processor, a training set, wherein the training set includes performance data, the plurality of state variable inputs, the plurality of biometric inputs, and the modulation parameter for each of the plurality of biometric inputs;
training, by the processor, a machine learning model based on the training set to generate a trained performance predictor;
receiving, by the processor:
the trained performance predictor;
biometric inputs predicted by following the slope and intercept of each modulation parameter to a prediction time; and
cognitive model state variable predictions made by progressing the model forward to the prediction time;

generating, by the processor, a prediction of ability to perform the task at the prediction time, generated by the trained performance predictor based on the predicted biometric inputs;
determining, by the processor, that the ability to perform the task is below a threshold level; and
activating, by the processor, an automated system configured to directly control the vehicle to change movement in response to determining that the ability to perform the task is below the threshold level.

14. The system of claim 13 , wherein the plurality of sensors sense physiological data during the performance of the task.

15. The system of claim 13 , wherein the plurality of sensors sense and detect data from an environment during the performance of the task.

16. A non-transitory computer readable medium containing program instructions that, when executed by a processor, cause the processor to generate a prediction of ability to perform a task in a future time step by:
receiving, by the processor, performance data corresponding to a level of performance of the task;
receiving, by the processor, a plurality of biometric inputs computed based on physiological data sensed by a first plurality of sensors during the performance of the task;
identifying, by the processor, a numerical relationship between the performance data and the plurality of biometric inputs;
generating, by the processor, a modulatio

Assignees

• Hrl Lab Llc

Inventors

• Howard Michael D
• Pilly Praveen K

Classifications

• G06N3/09

  Supervised learning · CPC title

• G09B7/00

  Electrically-operated teaching apparatus or devices working with questions and answers (mechanically operated G09B3/00; computing arrangements G06F) · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G09B19/0069

  Engineering, e.g. mechanical, electrical design (computer engineering G09B19/0053) · CPC title

• G09B19/167 Primary

  Control of land vehicles · CPC title