Saliency based awareness modeling

Patent metadata

Field	Value
Publication number	US-9747812-B2
Application number	US-201414521167-A
Country	US
Kind code	B2
Filing date	Oct 22, 2014
Priority date	Oct 22, 2014
Publication date	Aug 29, 2017
Grant date	Aug 29, 2017

Abstract

In one or more embodiments, driver awareness may be calculated, inferred, or estimated utilizing a saliency model, a predictive model, or an operating environment model. An awareness model including one or more awareness scores for one or more objects may be constructed based on the saliency model or one or more saliency parameters associated therewith. A variety of sensors or components may detect one or more object attributes, saliency, operator attributes, operator behavior, operator responses, etc. and construct one or more models accordingly. Examples of object attributes associated with saliency or saliency parameters may include visual characteristics, visual stimuli, optical flow, velocity, movement, color, color differences, contrast, contrast differences, color saturation, brightness, edge strength, luminance, a quick transient (e.g., a flashing light, an abrupt onset of a change in intensity, brightness, etc.).

First claim

What is claimed is:

1. A system for saliency based awareness modeling, comprising:
a sensor component detecting one or more objects within an operating environment and one or more object attributes for one or more of the objects, wherein one or more of the object attributes are associated with saliency of one or more of the objects;
a monitoring component detecting one or more operator attributes of an operator of a vehicle;
a modeling component constructing:
a saliency model for one or more of the objects based on one or more of the attributes associated with saliency of one or more of the objects; and
an awareness model for one or more of the objects based on the saliency model and one or more of the operator attributes; and

a scoring component assigning one or more awareness scores to one or more objects of the awareness model based on the saliency model and one or more of the operator attributes,
wherein the sensor component, the monitoring component, the modeling component, or the scoring component is implemented via a processing unit.

2. The system of claim 1 , comprising an electronic control unit (ECU) receiving one or more operator responses or operator behavior associated with the operator of the vehicle.

3. The system of claim 2 , wherein the modeling component constructs the awareness model based on one or more of the operator responses.

4. The system of claim 2 , comprising a database component housing baseline operator response information, wherein the modeling component constructs the awareness model based on a comparison between the baseline operator response information and one or more of the operator responses.

5. The system of claim 1 , comprising a notification component generating one or more notifications based on one or more awareness scores for one or more of the objects.

6. The system of claim 5 , comprising a management component controlling a timing, a color, or a size of one or more of the notifications.

7. The system of claim 1 , wherein the sensor component comprises an image capture device, a radar sensor, a light detection and ranging (LIDAR) sensor, a laser sensor, a video sensor, or a movement sensor.

8. The system of claim 1 , wherein the monitoring component comprises an image capture sensor, a motion sensors, an eye tracking unit, an infrared sensor, an infrared illuminator, or a depth sensor.

9. The system of claim 1 , wherein one or more of the object attributes comprises velocity, color, contrast, color saturation, brightness, or a detected transient for one or more of the objects.

10. The system of claim 1 , wherein one or more of the operator attributes comprises eye movement, head movement, focus, facial trajectory, eye gaze trajectory, or gaze distribution.

11. A method for saliency based awareness modeling, comprising:
detecting, using a sensor component, one or more objects within an operating environment;
detecting one or more object attributes for one or more of the objects, wherein one or more of the object attributes are associated with saliency of one or more of the objects;
detecting one or more operator attributes of an operator of the vehicle;
receiving one or more operator responses provided by the operator of the vehicle;
constructing a saliency model for one or more of the objects based on one or more of the attributes associated with saliency of one or more of the objects;
constructing an awareness model for one or more of the objects based on the saliency model, one or more of the operator responses, and one or more of the operator attributes; and
assigning one or more awareness scores to one or more objects of the awareness model based on the saliency model, one or more of the operator responses, and one or more of the operator attributes,
wherein the detecting, the receiving, the constructing, or the assigning is implemented via a processing unit.

12. The method of claim 11 , comprising constructing the awareness model based on a comparison between baseline operator response information and one or more of the operator responses.

13. The method of claim 11 , comprising constructing the awareness model based on a comparison between baseline object attribute information and one or more of the object attributes.

14. The method of claim 11 , comprising constructing the awareness model based on a comparison between baseline operator attribute information and one or more of the operator attributes.

15. The method of claim 11 , comprising rendering one or more notifications based on one or more awareness scores for one or more of the objects during navigation from an origin location to a destination location.

16. The method of claim 15 , comprising managing one or more aspects of one or more of the notifications.

17. A system for saliency based awareness modeling, comprising:
a sensor component detecting one or more objects within an operating environment and one or more object attributes for one or more of the objects, wherein one or more of the object attributes are associated with saliency of one or more of the objects;
a monitoring component detecting one or more operator attributes of an operator of a vehicle;
a modeling component constructing:
a saliency model for one or more of the objects based on one or more of the attributes associated with saliency of one or more of the objects; and
an awareness model for one or more of the objects based on the saliency model and one or more of the operator attributes;

a scoring component assigning one or more awareness scores to one or more objects of the awareness model based on the saliency model and one or more of the operator attributes; and
a notification component generating one or more notifications based on one or more awareness scores for one or more of the objects,
wherein the sensor component, the monitoring component, the modeling component, the scoring component, or the notification component is implemented via a processing unit.

18. The system of claim 17 , wherein the sensor component is a gaze detection device tracking eye movement or gaze distribution.

19. The system of claim 17 , comprising an electronic control unit (ECU) determining a number of attention demanding objects based on user interaction with one or more subunits of the ECU.

20. The system of claim 19 , wherein the modeling component constructs the awareness model based on the number of attention demanding objects.

Assignees

• Honda Motor Co Ltd

Inventors

• Misu Teruhisa

Classifications

• B60W50/14

  Means for informing the driver, warning the driver or prompting a driver intervention · CPC title

• B60W2050/146

  Display means · CPC title

• B60W2050/143

  Alarm means · CPC title

• G09B9/04 Primary

  for teaching control of land vehicles · CPC title

• B60W50/085

  Changing the parameters of the control units, e.g. changing limit values, working points by control input · CPC title