Depth reconstruction using plural depth capture units

Patent metadata

Field	Value
Publication number	US-9536312-B2
Application number	US-201113107986-A
Country	US
Kind code	B2
Filing date	May 16, 2011
Priority date	May 16, 2011
Publication date	Jan 3, 2017
Grant date	Jan 3, 2017

Abstract

A depth construction module is described that receives depth images provided by two or more depth capture units. Each depth capture unit generates its depth image using a structured light technique, that is, by projecting a pattern onto an object and receiving a captured image in response thereto. The depth construction module then identifies at least one deficient portion in at least one depth image that has been received, which may be attributed to overlapping projected patterns that impinge the object. The depth construction module then uses a multi-view reconstruction technique, such as a plane sweeping technique, to supply depth information for the deficient portion. In another mode, a multi-view reconstruction technique can be used to produce an entire depth scene based on captured images received from the depth capture units, that is, without first identifying deficient portions in the depth images.

First claim

What is claimed is:

1. A storage device for storing computer readable instructions, the computer readable instructions providing a depth construction module executed by one or more processing devices, the computer readable instructions performing a method comprising:
receiving at least a first depth image from a first depth capture unit and a second depth image from a second depth capture unit,
the first depth image and the second depth image being generated, respectively, by the first depth capture unit and the second depth capture unit, in response to:
projecting, using a first projector, a first projected pattern onto an object;
projecting, using a second projector, a second projected pattern onto the object;
capturing, using a first camera, a first captured image of the object;
capturing, using a second camera, a second captured image of the object;
providing the first depth image based on the first projected pattern and the first captured image; and
providing the second depth image based on the second projected pattern and the second captured image;

identifying at least one deficient portion in at least one of the first depth image and the second depth image;
generating depth information for said at least one deficient portion using a multi-view reconstruction technique, the multi-view reconstruction technique including establishing a plurality of planes that provide a plurality of hypothetical surface positions for the object and for each hypothetical surface position on each plane, deriving parameters which indicate an extent to which respective projected patterns impinge the hypothetical surface position; and
merging the depth information for said at least one deficient portion with at least the first depth image and the second depth image to produce a depth scene that provides a depth map of the object based on said parameters.

2. The storage device of claim 1 , wherein said at least one deficient portion is from a part of the object on which at least the first projected pattern overlaps the second projected pattern.

3. The storage device of claim 1 , wherein said at least one deficient portion corresponds to an omission of at least one depth value in at least one of the first depth image and the second depth image.

4. The storage device of claim 1 , wherein the method further comprises:
receiving first confidence scores from the first depth capture unit associated with depths in the first depth image; and
receiving second confidence scores from the second depth capture unit associated with depths in the second depth image,
wherein said at least one deficient portion is assessed based the first confidence scores and the second confidence scores.

5. The storage device of claim 4 , wherein the first confidence scores comprise cross-correlation scores produced by matching the first captured image with the first projected pattern, and the second confidence scores comprise cross-correlation scores produced by matching the second captured image with the second projected pattern.

6. The storage device of claim 1 , wherein the method further comprises:
receiving the first captured image from the first depth capture unit and the second captured image from the second depth capture unit;
generating first confidence scores by matching the first captured image with the first projected pattern;
generating second confidence scores by matching the second captured image with the second projected pattern, and
wherein said at least one deficient portion is assessed based on the first confidence scores and the second confidence scores.

7. The storage device of claim 1 , wherein the multi-view reconstruction technique jointly takes into consideration plural captured images provided by plural depth capture units.

8. The storage device of claim 1 , wherein the multi-view reconstruction technique comprises a plane sweeping technique.

9. The storage device of claim 1 , wherein said generating of the depth information for said at least one deficient portion leverages a constraint relationship that affects at least the first captured image and the second captured image.

10. The storage device of claim 9 , wherein the constraint relationship includes:
a first assertion that a first portion of the first captured image will match a second portion of the second captured image when the first portion and the second portion represent a same part of the object; and
a second assertion that each of the first portion and the second portion will match an aggregation of projected patterns which impinges the part of the object.

11. A device comprising:
a processor; and
executable instructions operable by the processor, the executable instructions comprising a method for depth reconstruction, the method comprising:
receiving at least a first depth image from a first depth capture unit and a second depth image from a second depth capture unit,
the first depth image and the second depth image being generated, respectively, by the first depth capture unit and the second depth capture unit, in response to:
projecting, using a first projector, a first projected pattern onto an object;
projecting, using a second projector, a second projected pattern onto the object;
capturing, using a first camera, a first captured image of the object;
capturing, using a second camera, a second captured image of the object;
providing the first depth image based on the first projected pattern and the first captured image; and
providing the second depth image based on the second projected pattern and the second captured image;

identifying at least one deficient portion in at least one of the first depth image and the second depth image;
generating depth information for the at least one deficient portion using a multi-view reconstruction technique, the multi-view reconstruction technique including establishing a plurality of planes that provide a plurality of hypothetical surface positions for the object and for each hypothetical surface position on each plane, deriving parameters which indicate an extent to which respective projected patterns impinge the hypothetical surface position; and
merging the depth information for the at least one deficient portion with at least the first depth image and the second depth image to produce a depth scene that provides a depth map of the object based on said parameters.

12. The device of claim 11 , wherein the method further comprises:
receiving first confidence scores from the first depth capture unit associated with depths in the first depth image; and
receiving second confidence scores from the second depth capture unit associated with depths in the second depth image, wherein the at least one deficient portion is assessed based the first confidence scores and the second confidence scores.

13. The device of claim 12 , wherein the first confidence scores comprise cross-correlation scores produced by matching the first captured image with the first projected pattern, and the second confidence scores comprise cross-correlation scores produced by matching the second captured image with the second projected pattern.

14. The device of claim 11 , wherein the method further comprises:
receiving the first captured image from the first depth capture unit and the second captured image from the second depth capture unit;
generating first confidence scores by matching the first captured image with the first projected pattern; and
generating second confidence scores by matching the second captured image with the second projected pattern, wherein the at least one deficient portion is assessed based on the first confid

Assignees

• Zhang Cha
• Zhu Wenwu
• Zhang Zhengyou
• Chou Philip A
• Microsoft Corp

Inventors

• Zhang Cha
• Zhu Wenwu
• Zhang Zhengyou
• Chou Philip A

Classifications

• G06V10/145

  Illumination specially adapted for pattern recognition, e.g. using gratings · CPC title

• G06K9/2036

  Physics · mapped topic

• G06T2207/10048

  Infrared image · CPC title

• G06T7/0057 Primary

  Physics · mapped topic

• G06T2207/10152

  Varying illumination · CPC title