Low-power always-on face detection, tracking, recognition and/or analysis using events-based vision sensor

Patent metadata

Field	Value
Publication number	US-9986211-B2
Application number	US-201715400914-A
Country	US
Kind code	B2
Filing date	Jan 6, 2017
Priority date	Sep 30, 2014
Publication date	May 29, 2018
Grant date	May 29, 2018

Abstract

Techniques disclosed herein utilize a vision sensor that integrates a special-purpose camera with dedicated computer vision (CV) computation hardware and a dedicated low-power microprocessor for the purposes of detecting, tracking, recognizing, and/or analyzing subjects, objects, and scenes in the view of the camera. The vision sensor processes the information retrieved from the camera using the included low-power microprocessor and sends “events” (or indications that one or more reference occurrences have occurred, and, possibly, associated data) for the main processor only when needed or as defined and configured by the application. This allows the general-purpose microprocessor (which is typically relatively high-speed and high-power to support a variety of applications) to stay in a low-power (e.g., sleep mode) most of the time as conventional, while becoming active only when events are received from the vision sensor.

First claim

What is claimed is:

1. A device comprising:
dedicated computer vision (CV) computation hardware configured to receive sensor data from a sensor array comprising more than one sensor pixel and capable of computing one or more CV features using readings from neighboring sensor pixels of the sensor array; and
a first processing unit communicatively coupled with the dedicated CV computation hardware and configured to:
process signals resulting from operations based on the one or more computed CV features, and
in response to the processing of the signals resulting from the operations based on the one or more computed CV features, generate an event indicating a reference occurrence; and

a second processing unit communicatively coupled with the first processing unit and configured to execute application software in response to receiving the event.

2. The device of claim 1 , wherein the event is configured to indicate the reference occurrence comprises at least one of:
a coming into view of a human face,
a coming into view of a human body,
an emotion expressed on a human face,
coming into view of an non-human animal face,
coming into view of a non-human animal body,
coming into view of a human hand,
a hand gesture,
a coming into view of a reference object,
a change from an indoor environment to an outdoor environment,
a reference movement,
rapid movement in a scene indicating a fall,
motion toward an object indicating a risk of collision,
movement or objects in a scene indicating danger, or
any combination thereof.

3. The device of claim 1 , wherein the event is configured to indicate the reference occurrence comprises object detection.

4. The device of claim 1 , wherein the dedicated CV computation hardware is configured to provide Local Binary Patterns (LBPs).

5. The device of claim 1 , further comprising a classifier configured to detect a presence of a reference object in a subset of the sensor data, wherein the operations based on the one or more computed CV features comprise operations performed by the classifier, the reference occurrence being associated with the reference object.

6. The device of claim 5 , wherein the first processing unit is configured to receive an indication from the classifier of the presence of the reference object when the presence of the reference object is detected by the classifier.

7. The device of claim 5 , wherein the classifier comprises a cascade classifier comprising software executed by the first processing unit.

8. The device of claim 1 , wherein the dedicated CV computation hardware comprises an integrated circuit.

9. The device of claim 1 , wherein the first processing unit is further configured to disable at least part of the dedicated CV computation hardware to generate at least one lower-power optical sensor reading indicative of ambient light sensing, a proximity detection, a proximity to a reference object, a motion detection, or any combination thereof for use by the second processing unit.

10. The device of claim 1 , wherein the first processing unit is configured to disable at least part of the dedicated CV computation hardware while operating in a lower-power mode, detect a scene change in the lower-power mode, and to activate the dedicated CV computation hardware based on detection of the scene change.

11. The device of claim 1 , wherein the first processing unit is configure to include, in the event, data associated with the reference occurrence.

12. The device of claim 11 , wherein the second processing unit is configured to execute the application software in response to receiving the event comprises the second processing unit utilizing the data associated with the reference occurrence while executing the application software.

13. The device of claim 1 , wherein the first processing unit is configured to:
perform a higher-power operation as at least part of the operations based on the one or more computed CV features, the higher-power operation consuming more power than a lower-power operation, and
provide a parameter for the lower-power operation based on the higher-power operation.

14. The device of claim 13 , wherein the lower-power operation includes one or more from the list comprising: ambient light sensing, proximity detection, motion detection, or change detection.

15. A device comprising:
a sensor array comprising more than one sensor pixel;
dedicated computer vision (CV) computation hardware capable of receiving sensor data from the sensor array and computing CV features using readings from neighboring sensor pixels of the sensor array;
a first processing unit communicatively coupled with the dedicated CV computation hardware and configured to:
determine, from one or more signals resulting from operations based on the one or more computed CV features that a face has been detected, and
in response to the determination, generate an object-detected event; and

a second processing unit communicatively coupled with the first processing unit and configured to execute application software in response to receiving the object-detected event.

16. The device of claim 15 , wherein the first processing unit is configured to communicate the object-detected event to the second processing unit while the second processing unit is operating in a low-power mode.

17. The device of claim 15 , wherein the dedicated CV computation hardware is configured to receive raw sensor data from the sensor array and no image signal processing circuitry is disposed between the sensor array and the dedicated CV computation hardware.

18. The device of claim 15 , further comprising peripheral circuitry configured to provide at least one of:
a timing operation,
a focusing operation,
an auto-exposure correction operation,
object detection,
object recognition,
storing a scanning window,
an event-queuing and/or processing operation,
analog processing,
analog-to-digital conversion,
an integration operation,
CV feature computation,
a cascade-classifier-based classification,
a histogram-based classification, or
memory buffering, or
any combination thereof.

19. A method for sensing dynamic scene-based occurrences with an apparatus, the method comprising:
receiving, with dedicated computer vision (CV) computation hardware, sensor data from a sensor array comprising more than one sensor pixel;
computing, with the dedicated CV computation hardware, one or more CV features using readings from neighboring sensor pixels of the sensor array;
processing, with a first processing unit, signals resulting from operations based on the one or more computed CV features;
generating an event in response to the processing of the signals resulting from the operations based on the one or more computed CV features by the first processing unit, the event indicating a reference occurrence for a second processing unit; and
executing, with a second processing unit, application software in response to the event configured.

20. The method of claim 19 , wherein the reference occurrence is one or more of:
a coming into view of a human face,
a coming into view of a human body,
an emotion expressed on a human face,
coming into view of an non-human animal face,
coming into view of a non-human animal body,
coming into view of a human hand,
a hand gesture,
a coming into view of a reference object,
a change from an indoor environment to an outdoor environment,
a reference movement,
rapid movement in a scene indicating a fall,
motion toward an ob

Assignees

• Qualcomm Inc

Inventors

• Gousev Evgeni
• Govil Alok
• Maitan Jacek
• Rasquinha Nelson
• Rangan Venkat

Classifications

• G06V40/10 Primary

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• H04N7/183 Primary

  for receiving images from a single remote source · CPC title

• H04N23/651

  for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N23/61

  based on recognised objects · CPC title