Monitoring activity with depth and multi-spectral camera

Patent metadata

Field	Value
Publication number	US-11245875-B2
Application number	US-201916248409-A
Country	US
Kind code	B2
Filing date	Jan 15, 2019
Priority date	Jan 15, 2019
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

A camera system is configured to automatically monitor an area. Depth image(s) of the area are acquired based on active IR light emitted by the camera system and reflected from the area to a sensor array of the camera system. The depth image(s) are computer analyzed to identify a human subject. For each spectral illuminator of the camera system, spectral light image(s) of the area are acquired based on active spectral light in the spectral light sub-band of the spectral illuminator reflected from the area to the sensor array. The spectral light image(s) for the spectral illuminators are computer analyzed to identify an interaction between the human subject and an object in the area. In response to identifying the interaction between the human subject and the object in the area, an action to be performed for the object in the area is computer issued.

First claim

The invention claimed is:

1. A camera system configured to automatically monitor an area, the camera system comprising:
a sensor array including a plurality of sensors;
an infrared (IR) illuminator configured to emit active IR light in an IR light sub-band;
a plurality of spectral illuminators, each spectral illuminator configured to emit active spectral light in a different spectral light sub-band;
one or more logic machines; and
one or more storage machines holding instructions executable by the one or more logic machines to:
address the plurality of sensors of the sensor array without active illumination from the IR illuminator and without active illumination from the plurality of spectral illuminators to acquire one or more images of the area;
computer analyze the one or more images to identify an above-threshold motion;
in response to identifying the above-threshold motion:
activate the IR illuminator to illuminate at least a first portion of the area with the active IR light, and
address the plurality of sensors of the sensor array to acquire one or more depth images of the first portion of the area based on active IR light reflected from the first portion of the area,

computer analyze the one or more depth images to identify a human subject;
in response to identifying the human subject, for each spectral illuminator of the plurality of spectral illuminators:
activate the spectral illuminator to illuminate a second portion of the area with the active spectral light in only a spectral light sub-band of the spectral illuminator,
address the plurality of sensors of the sensor array to acquire one or more spectral light images of the second portion of the area based on active spectral light in the spectral light sub-band of the spectral illuminator reflected from the second portion of the area and without filtering light in other spectral sub-bands reflected from the second portion of the area to the plurality of sensors of the sensor array,
computer analyze at least the one or more spectral light images to identify an interaction between the human subject and an object in the area, and
in response to identifying the interaction between the human subject and the object, computer issue an action to be performed for the object in the area.

2. The camera system of claim 1 , wherein the one or more storage machines hold instructions executable by the one or more logic machines to: computer recognize that the human subject is a shopper, and wherein the action includes a retail action of computer associating the object in the area with the shopper.

3. The camera system of claim 1 , wherein the action includes a retail action of sending to a retail computer a notification to restock the object in the area.

4. The camera system of claim 1 , wherein the interaction includes the human subject picking up the object and then replacing the object in the area, and wherein the action includes a retail action of sending to a retail computer a notification to inspect the object.

5. The camera system of claim 1 , wherein the one or more storage machines hold instructions executable by the one or more logic machines to: computer analyze the one or more depth images and/or the one or more spectral light images to determine a volume of the object, and wherein the action includes a retail action of associating a cost with the object based on a weight of the object that is derived from the determined volume of the object.

6. The camera system of claim 1 , further comprising:
a peripheral pressure sensor configured to generate a pressure signal that indicates whether or not the object is positioned on the peripheral pressure sensor; and
wherein the interaction between the human subject and the object is further identified based on the pressure signal.

7. The camera system of claim 6 , wherein the action includes a retail action of associating a cost with the object based on a weight of the object that is derived from the pressure signal.

8. The camera system of claim 1 , wherein the one or more storage machines hold instructions executable by the one or more logic machines to: measure a spectral signature of the object based on the one or more spectral light images and the one or more depth images; and computer identify the object based on the measured spectral signature.

9. The camera system of claim 1 , wherein the sensor array, the IR illuminator, and the plurality of spectral illuminators are positioned above the area.

10. The camera system of claim 1 , wherein the sensor array, the IR illuminator, and the plurality of spectral illuminators are incorporated into a head-mounted device wearable by the human subject.

11. The camera system of claim 1 , wherein the sensor array, the IR illuminator, and the plurality of spectral illuminators are incorporated into a shopping cart or basket.

12. A method for automatically monitoring an area, the method comprising:
addressing a plurality of sensors of a sensor array of a camera without active illumination from an IR illuminator of the camera and without active spectral illumination from a plurality of spectral illuminators of the camera to acquire one or more images of the area;
computer analyzing the one or more images to identify an above-threshold motion;
in response to identifying the above-threshold motion:
activating the IR illuminator to illuminate at least a first portion of the retail shopping area with the active IR light, and
addressing the plurality of sensors of the sensor array to acquire one or more depth images of the first portion of the area based on active IR light reflected from the first portion of the area,

computer analyzing the one or more depth images to identify a human subject;
in response to identifying the human subject, for each spectral illuminator of the plurality of spectral illuminators:
activating the spectral illuminator to illuminate a second portion of the area with the active spectral light only in a spectral light sub-band of the spectral illuminator,
addressing the plurality of sensors of the sensor array to acquire one or more spectral light images of the second portion of the area based on active spectral light in the spectral light sub-band of the spectral illuminator reflected from the second portion of the area and without filtering light in other spectral sub-bands reflected from the second portion of the area to the plurality of sensors of the sensor array,
computer analyzing at least the one or more spectral light images to identify an interaction between the human subject and an object in the area, and
in response to identifying the interaction between the human subject and the object, computer issuing an action to be performed for the object in the area.

13. The method of claim 12 , further comprising:
computer recognizing that the human subject is a shopper, and wherein the action includes a retail action of computer associating the object in the area with the shopper.

14. The method of claim 12 , wherein the action includes a retail action of sending to a retail computer a notification to restock the object for purchase.

15. The method of claim 12 , wherein the interaction includes the human subject picking up the object and then replacing the object in the area, and wherein the action includes a retail action of sending to a retail computer a notification to inspect the object.

16. The method of claim 12 , further comprising:
computer analyzing the one or more depth images and/or the one or more spectral light images to determine a volume of the object, and wherein the action inclu

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Ortiz Egea Sergio
• Ali Riaz Imdad
• Fenton Michael Scott
• Akkaya Onur Can

Classifications

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06V10/143 Primary

  Sensing or illuminating at different wavelengths · CPC title

• H04N7/181 Primary

  for receiving images from a plurality of remote sources · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title