System for background subtraction with 3D camera

Patent metadata

Field	Value
Publication number	US-10325360-B2
Application number	US-201715785407-A
Country	US
Kind code	B2
Filing date	Oct 16, 2017
Priority date	Aug 30, 2010
Publication date	Jun 18, 2019
Grant date	Jun 18, 2019

Abstract

A system for background image subtraction includes a computing device coupled with a 3D video camera, a processor of the device programmed to receive a video feed from the camera containing images of one or more subject that include depth information. The processor, for an image: segments pixels and corresponding depth information to identify foreground regions associated with a target user.

First claim

The invention claimed is:

1. A mobile computing device comprising:
at least one color sensor configured to obtain color frames of a scene having a target subject, the color frames having a color sensor viewpoint;
a depth sensor having an infrared illuminator and an infrared sensor, configured to generate depth data of the scene from a depth sensor viewpoint;
a processor and memory configured to:
a) identify a set of potential foreground pixel locations by processing the depth data;
b) detect one or more foreground connected components;
c) determine a near mode based on an average depth of the one or more foreground connected components;
d) responsive to determining the near mode, form a reduced set of potential foreground pixel locations by removing some pixel locations from the set of potential foreground pixel locations that are below a center of the one or more connected foreground components;
e) process the reduced set of potential foreground pixel locations to generate aligned foreground depth data associated with a target user by warping the reduced set of potential foreground pixel locations from the depth sensor viewpoint to the color sensor viewpoint; and,
f) construct at least one new color image based on the aligned foreground depth data and the color frames from the at least one color sensor.

2. The mobile computing device of claim 1 , wherein the processor and memory are further configured to display the at least one new color image as frames in a video stream.

3. The mobile computing device of claim 2 , wherein the video stream is displayed in real-time.

4. The mobile computing device of claim 1 , wherein identifying a set of potential foreground pixel locations by processing the depth data comprises, for each of a plurality of pixels, each pixel having a depth along a Z axis from the depth sensor, identifying a pixel as a potential foreground pixel if its respective depth is less than a predetermined threshold distance.

5. The mobile computing device of claim 1 , wherein the near mode is determined according to an average depth of a largest connected foreground component.

6. The mobile computing device of claim 1 , wherein the set of potential foreground pixels is further reduced according to color information and background history (BGH) information associated with the potential foreground pixels.

7. The mobile computing device of claim 1 , further comprising, assigning depth values to each foreground pixel of the color frame having unknown depth values by interpolating the aligned foreground depth data.

8. The mobile computing device of claim 1 , wherein detecting one or more foreground connected components comprises labeling as foreground connected components pixels that are (i) adjacent to each other, (ii) identified as potential foreground pixels, and (iii) have adjacent depth values within a predetermined threshold.

9. The system of claim 8 , wherein foreground connected components are further limited to pixels having a depth value less than a predetermined threshold distance and an intensity value greater than a predetermined threshold intensity.

10. The system of claim 1 , wherein the at least one new color image further comprises at least one image from a second video.

Assignees

• Univ Illinois

Inventors

• Nguyen Quang H
• Do Minh N
• Patel Sanjay J
• Dabbelt Daniel P
• Lin Dennis J

Classifications

• G06V10/28

  Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns · CPC title

• G06K9/38

  Physics · mapped topic

• H04N5/272

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T2207/10024

  Color image · CPC title