Techniques for assessing group level cognitive states

Patent metadata

Field	Value
Publication number	US-10216983-B2
Application number	US-201615370736-A
Country	US
Kind code	B2
Filing date	Dec 6, 2016
Priority date	Dec 6, 2016
Publication date	Feb 26, 2019
Grant date	Feb 26, 2019

Abstract

A security monitoring technique includes receiving data related to one or more individuals from one or more cameras in an environment. Based on the input data from the cameras, agent-based simulators are executed that each operate to generate a model of behavior of a respective individual, wherein an output of each model is symbolic sequences representative of internal experiences of the respective individual during simulation. Based on the symbolic sequences, a subsequent behavior for each of the respective individuals is predicted when the symbolic sequences match a query symbolic sequence for a query behavior.

First claim

The invention claimed is:

1. A method, comprising:
receiving data related to one or more individuals from one or more cameras in an environment;
executing one or more agent-based simulators that each operate to generate a model of behavior of a respective individual, wherein an output of each model is symbolic sequences representative of internal experiences of the respective individual during simulation; and
predicting a subsequent behavior for each of the respective individuals when the symbolic sequences match a query symbolic sequence for a query behavior;
wherein each model uses particle filtering and each particle includes recurrent neural networks that iteratively estimates temporal evolution of the symbolic sequences based on the data.

2. The method of claim 1 , wherein particles that include similar symbolic sequences are allowed to transition to the next iteration to predict a next set of internal symbols of the symbolic sequences.

3. The method of claim 1 , wherein particles that do not include similar symbolic sequences are terminated.

4. The method of claim 1 , wherein the recurrent neural networks are used to predict the subsequent behavior based on the symbolic sequences.

5. The method of claim 4 , wherein the recurrent neural networks are seeded with random internal experience symbols initially.

6. The method of claim 5 , wherein the recurrent neural networks predict the subsequent behavior by sampling a next set of physical state symbols and comparing the next set of physical symbols to physical state symbols of the query symbolic sequence.

7. The method of claim 1 , wherein the symbolic sequences include stored graphics for character type, emotion, observed expressions, or some combination thereof.

8. The method of claim 7 , wherein the character type comprises sunshine, predator, stranger, depressed, or nervous, and the emotion comprises angry, frustrated, neutral, or happy.

9. The method of claim 1 , comprising performing an action when a certain behavior is predicted, wherein the action comprises sounding an alarm, calling emergency services, triggering an alert, sending a message, displaying an alert, or some combination thereof.

10. The method of claim 1 , wherein the one or more cameras comprise red, green, blue, depth (RGB+D) cameras that capture estimates of location and articulated body motion, and fixed cameras and pan tilt zoom (PTZ) cameras that capture facial imagery.

11. One or more tangible, non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to:
receive data related to one or more individuals from one or more cameras in an environment;
execute one or more agent based simulators that each model behavior of a respective individual and each output symbolic sequences representative of internal experiences of the respective individual during simulation; and
predict a subsequent behavior for each of the respective individuals when the symbolic sequences match a query symbolic sequence for a query behavior;
wherein each model uses particle filtering and each particle includes recurrent neural networks that iteratively estimates temporal evolution of the symbolic sequences based on the data.

12. The one or more computer-readable media of claim 11 , wherein particles that include similar symbolic sequences are allowed to transition to the next iteration to predict a next set of internal symbols of the symbolic sequences.

13. The one or more computer-readable media of claim 11 , wherein the recurrent neural networks are used to predict the subsequent behavior based on the symbolic sequences.

14. The one or more computer-readable media of claim 11 , wherein the computer instructions, when executed by the processor, cause the one or more processors to perform an action when a certain behavior is predicted, wherein the action comprises sounding an alarm, calling emergency services, triggering an alert, sending a message, displaying an alert, or some combination thereof.

15. A system, comprising:
one or more cameras that capture data related to a behavior of one or more individuals in an environment;
one or more computing devices comprising one or more processors that:
receive the data related to the behavior of one or more individuals from one or more cameras in an environment;
execute one or more agent based simulators that each model the behavior of a respective individual and each output symbolic sequences representative of internal experiences of the respective individual during simulation; and
predict a subsequent behavior for each of the respective individuals when the symbolic sequences match a query symbolic sequence for a query behavior; and

a display coupled to the one or more computing devices and configured to display an indication representative of the subsequent behavior;
wherein each model uses particle filtering and each particle includes recurrent neural networks that iteratively estimates temporal evolution of the symbolic sequences based on the data.

16. The system of claim 15 , wherein the one or more cameras comprise red, green, blue, depth (RGB+D) cameras that capture estimates of location and articulated body motion, and fixed cameras and pan tilt zoom (PTZ) cameras that capture facial imagery.

17. The system of claim 15 , wherein the one or more computing devices comprise a smartphone, a smartwatch, a tablet, a laptop computer, a desktop computer, a server in a cloud-based computing system, or some combination thereof.

18. The system of claim 15 , wherein the one or more processors perform an action when a certain subsequent behavior is predicted, the action comprising sounding an alarm, calling emergency services, triggering an alert, sending a message, displaying an alert, or some combination thereof.

Assignees

• Gen Electric

Inventors

• Tu Peter Henry
• Gao Tao
• Tu Jilin

Classifications

• G06V10/62 Primary

  relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F2218/16

  by matching signal segments · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title