Distinguishing foreground and background with inframed imaging

Patent metadata

Field	Value
Publication number	US-9955140-B2
Application number	US-201514645326-A
Country	US
Kind code	B2
Filing date	Mar 11, 2015
Priority date	Mar 11, 2015
Publication date	Apr 24, 2018
Grant date	Apr 24, 2018

Abstract

An initial candidate foreground region is identified within an infrared image that includes pixels exhibiting infrared intensity values within a pre-defined range. A depth of surfaces within the initial candidate foreground region is estimated based on infrared intensity values the pixels of the initial candidate foreground region. The initial candidate foreground region is expanded to an expanded candidate foreground region based on a body-model estimate. The body model estimate is seeded with one or more of the initial candidate foreground region, the depth of surfaces, and/or a face of a human subject identified by facial recognition. Each pixel of the infrared image is identified as either a foreground pixel or a background pixel based on a distance of that pixel relative to the expanded candidate foreground region. Pixels identified as background pixels may be modified within a corresponding visible light image.

First claim

The invention claimed is:

1. An imaging method, comprising:
obtaining an infrared image captured via an infrared camera;
identifying an initial candidate foreground region within the infrared image that includes pixels exhibiting infrared intensity values within a pre-defined range;
estimating depth values of surfaces within the initial candidate foreground region of the infrared image based on infrared intensity values of the pixels within the initial candidate foreground region of the infrared image; and
expanding the initial candidate foreground region to an expanded candidate foreground region based on a body-model estimate seeded with the initial candidate foreground region and the depth values of the surfaces estimated within the initial candidate foreground region.

2. The method of claim 1 , further comprising:
identifying each pixel of the infrared image as either a foreground pixel or a background pixel based on a distance of that pixel relative to the expanded candidate foreground region.

3. The method of claim 2 , further comprising:
obtaining a visible light image captured via a visible light camera that is time-registered and spatially-registered with the infrared image; and
for some or all of the pixels of the infrared image identified as background pixels, modifying corresponding background pixels in the visible light image.

4. The method of claim 3 , further comprising:
obtaining a plurality of infrared images captured via the infrared camera, wherein the infrared image is one of the plurality of infrared images;
obtaining a plurality of visible light images captured via the visible light camera, wherein each visible light image is time-registered and spatially-registered with a corresponding infrared image of the plurality of infrared images;
for each infrared image of the plurality of infrared images, and for some or all of the pixels of that infrared image identified as background pixels, modifying corresponding background pixels in a visible light image that is time-registered and spatially registered with that infrared image; and
outputting the plurality of visible light images as a visible light video with each visible light image having modified background pixels.

5. The method of claim 2 , wherein each pixel located within the expanded candidate foreground region is identified as a foreground pixel, and some or all of the pixels located outside of the expanded candidate foreground region are identified as background pixels.

6. The method of claim 2 , further comprising:
weighting each pixel as either a foreground pixel or a background pixel based on a distance of that pixel from the expanded candidate foreground region;
identifying each pixel of the infrared image as either a foreground pixel or a background pixel based, at least in part, on a weighting assigned to that pixel; and
wherein the distance is a geodesic distance that follows a smallest gradient in light intensity among pixels located between each pixel and a closest pixel of the expanded candidate foreground region.

7. The method of claim 1 , further comprising:
weighting each pixel as either a foreground pixel or a background pixel based on a prior determination for that pixel in a previous frame; and
identifying each pixel of the infrared image as either a foreground pixel or a background pixel based, at least in part, on a weighting assigned to that pixel.

8. The method of claim 1 , wherein the pre-defined range includes infrared intensity values corresponding to human skin located within a depth range from the infrared camera that is illuminated by an infrared illumination condition.

9. The method of claim 8 , wherein the pre-defined range is obtained by previous training of a machine learning algorithm on one or more human training subjects located within the depth range from an infrared training camera that is illuminated by the infrared illumination condition.

10. The method of claim 1 , wherein identifying the initial candidate foreground region includes comparing infrared intensity values of individual pixels of the infrared image to the pre-defined range to obtain a set of pixels that exhibit infrared intensity values within the pre-defined range.

11. The method of claim 10 , wherein the initial candidate foreground region is one of a plurality of initial candidate foreground regions, in which each of the plurality of initial candidate foreground regions includes a threshold quantity or threshold proportion of pixels that exhibit infrared intensity values within the pre-defined range.

12. The method of claim 1 , further comprising:
applying facial recognition to the infrared image to identify a face of a human subject;
measuring one or more infrared intensity values from the face of the human subject; and
setting the pre-defined range based on the one or more infrared intensity values measured from the face of the human subject.

13. An imaging method, comprising:
obtaining an infrared image of an infrared illuminated scene captured via an infrared camera;
identifying an initial candidate foreground region within the infrared image that includes pixels exhibiting infrared intensity values within a pre-defined range;
estimating depth values of surfaces within the initial candidate foreground region of the infrared image based on infrared intensity values of the pixels within the initial candidate foreground region of the infrared image;
identifying a facial search region that is sized based on the depth values of the surfaces estimated within the initial candidate foreground region;
applying facial recognition initially constrained to the facial search region to identify a face of a human subject;
expanding the initial candidate foreground region to an expanded candidate foreground region based on a body-model estimate seeded with the face of the human subject; and
identifying each pixel of the infrared image as either a foreground pixel or a background pixel based on a distance of that pixel relative to the expanded candidate foreground region;
obtaining a visible light image that is registered with the infrared image; and
for some or all of the pixels of the infrared image identified as background pixels, blurring registered background pixels in the visible light image.

14. The method of claim 13 , further comprising:
determining a weighting for each pixel being a foreground pixel based on a distance of that pixel relative to the expanded candidate foreground region;
wherein identifying each pixel as either a foreground pixel or a background pixel is based on the weighting determined for that pixel; and
wherein the weighting determined for each pixel being a foreground pixel is greater if that pixel is closer to the expanded candidate foreground region and is smaller if that pixel is further from the expanded candidate foreground region.

15. The method of claim 14 , wherein the distance is a geodesic distance that follows a smallest gradient in light intensity among pixels located between each pixel and a closest pixel of the expanded candidate foreground region.

16. The method of claim 13 , wherein each pixel located within the expanded candidate foreground region is identified as a foreground pixel, and some or all of the pixels located outside of the expanded candidate foreground region are identified as background pixels.

17. The method of claim 13 , wherein the pre-defined range includes infrared intensity values corresponding to human skin located within a depth range from the infrared camera that is illuminated by an infrared illumina

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Rhemann Christoph
• Barsoum Emad
• Shen Yao
• Stachniak Simon P
• Izadi Shahram

Classifications

• G06V10/28

  Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns · CPC title

• G06V10/143

  Sensing or illuminating at different wavelengths · CPC title

• H04N5/33

  Transforming infrared radiation (cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20) · CPC title

• G06K9/00288

  Physics · mapped topic

• G06K9/38

  Physics · mapped topic