Systems and methods for providing visual allocation management

Patent metadata

Field	Value
Publication number	US-11688203-B2
Application number	US-202017120009-A
Country	US
Kind code	B2
Filing date	Dec 11, 2020
Priority date	Aug 19, 2016
Publication date	Jun 27, 2023
Grant date	Jun 27, 2023

Abstract

Systems and methods for managing visual allocation are provided herein that use models to determine states based on visual data and, based thereon, output feedback based on the determined states. Visual data is initially obtained by a visual allocation management system. The visual data includes eye image sequences of a person in a particular state, such as engaging in a task or activity. Visual features can be identified from the visual data, such that glance information including direction and duration can be calculated. The visual data, information derived therefrom, and/or other contextual data is input into the models, which correspond to states, to calculate probabilities that the particular state that the person is engaged in is one of the modeled states. Based on the state identified as having the highest probability, an optimal feedback, such as a warning or instruction, can be output to a connected devices, systems, or objects.

First claim

What is claimed is:

1. A system for managing visual allocation, comprising:
at least one memory operable to store one or more models, the one or more models including at least one of:
(1) a model corresponding to a plurality of candidate states; and
(2) a plurality of models in which each model corresponds to one of the plurality of candidate states; and at least one processor communicatively coupled to the at least one memory, the at least one processor being operable to:
receive visual data corresponding to a person engaging in an activity during a continuous period of time;
identify a sequence of glances from the visual data and, for each glance in the sequence of glances, identify corresponding glance information, the glance information including a glance direction;
classify each of the glances in the sequence of glances into a spatial region from among a set of predetermined spatial regions based on their respective glance information;
identify, based on the one or more models, the presence of one or more states of the person while engaging in the activity during the continuous period of time by inputting into each of the stored one or more models model input data including one or more of (1) the classification of the glances in the sequence of glances and (2) the glance information of the glances in the sequence of glances; and
output feedback based on the identified one or more states, wherein the one or more models correspond to one or more candidate states of the person that are not at least one of observable or detectable from the received visual data, and
wherein each candidate state and identified one or more states corresponds to at least one of an attention, awareness, emotion, a mental state, or a physical state of a person.

2. The system of claim 1 , wherein the processor is further operable to:
identify one or more eye image sequences in the visual data, the one or more eye image sequences including images of the eye region of the person engaging in the activity during the continuous period of time; and
extract visual features from each of the images of the one or more eye image sequences,
wherein the sequence of glances and corresponding glance information are identified from the one or more eye image sequences based on the extracted visual features.

3. The system of claim 1 , wherein, to identify the presence of one or more states at a given instance during the continuous period of time, the at least one processor is further operable to:
calculate, by the one or more models, respective probabilities of the presence of each of the plurality of candidate states based on model input data; and
select, as the identified one or more states, one or more of the plurality of candidate states having the highest respective calculated probabilities of presence at the given instance during the continuous period of time.

4. The system of claim 3 , wherein the output feedback includes one or more of:
(1) the probability of the presence of each of the plurality of candidate states during the continuous period of time;
(2) the identified one or more states having the highest respective calculated probabilities of presence during the continuous period of time; and
(3) instructions based on the identified one or more states of the person while engaged in the activity during the continuous period of time.

5. The system of claim 2 ,
wherein the at least one processor is further operable to identify, for each of the sequence of glances, one or more of a glance duration and a glance transition, the glance transition indicating the glance direction of a next glance in the sequence of glances,
wherein the glance information of each of the sequence of glances further includes the one or more of the respective glance duration and glance transition, and
wherein the identifying of the one or more states is further based on the one or more of the glance duration and the glance transition of each of the sequence of glances.

6. The system of claim 5 ,
wherein the at least one processor is further operable to identify one or more glance patterns from among the sequence of glances, based on the classification of each of the sequence of glances and/or the glance transitions of each of the sequence of glances, and
wherein the identifying the presence of the one or more states is further based on the glance transition patterns input into each of the one or more models.

7. The system of claim 1 , wherein the one or more models are Hidden Markov Models.

8. The system of claim 2 , wherein each of the one or more eye image sequences correspond to a single individual.

9. The system of claim 8 , wherein the one or more eye image sequences in the visual data are captured using one or more cameras.

10. The system of claim 1 , wherein the receiving of the visual data and outputting the result are performed in real-time.

11. The system of claim 1 ,
wherein the processor is further operable to receive contextual data from one or more communicatively coupled systems, and
wherein the identifying the presence of the one or more states is further based on the contextual data.

12. The system of claim 11 , wherein the contextual data includes at least one of environment data, surrounding data, and user data.

13. The system of claim 1 , wherein the at least one processor is further operable to:
train the one or more models using (1) visual data corresponding to a plurality of people engaging in the activity during continuous periods of time, and (2) state data including one or more states present in the plurality of people while engaging in the activity.

14. A vehicle comprising the system of claim 4 ,
wherein the person engaging in the activity is a driver engaging in the operating of the vehicle,
wherein the output feedback includes the instructions based on the identified one or more states, and
wherein the instructions include one or more of:
(1) instructions to manage the spatial attention or awareness of the driver of the vehicle,
(2) instructions causing an impact on control of the vehicle, and
(3) instructions to augment an exchange of information with at least one of the vehicle and the driver of the vehicle.

15. The vehicle of claim 14 , wherein the instructions to augment an exchange of information with at least one of the vehicle and the driver of the vehicle comprise instructions to suppress an exchange of information to and from the vehicle.

16. A method for managing visual allocation, comprising:
storing one or more models, the one or more models including at least one of:
(1) a model corresponding to a plurality of candidate states; and
(2) a plurality of models in which each model corresponds to one of the plurality of candidate states; and
receiving visual data corresponding to a person engaging in an activity during a continuous period of time;
identifying a sequence of glances from the visual data and, for each glance in the sequence of glances, identify corresponding glance information, the glance information including a glance direction;
classifying each of the glances in the sequence of glances into a spatial region selected from among a set of predetermined spatial regions, based on their respective glance information;
identifying, based on the one or more models, the presence of one or more states of the person while engaging in the activity during the continuous period of time, by inputting into each of the stored one or more models model input data including one or more of (1) the classification of the glances in the sequence of glances and (2) the glan

Assignees

• Massachusetts Inst Technology

Inventors

• Muñoz Delgado Andres Mauricio
• Reimer Bryan L
• Lee Joonbum
• Angell Linda Sala
• Seppelt Bobbie Danielle
• Mehler Bruce L
• Coughlin Joseph F

Classifications

• G06N5/04

  Inference or reasoning models · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06V40/19 Primary

  Sensors therefor · CPC title

• G06V10/85

  Markov-related models; Markov random fields · CPC title

• G06N20/00

  Machine learning · CPC title