System and method for active stereo depth sensing

Patent metadata

Field	Value
Publication number	US-10839539-B2
Application number	US-201815994509-A
Country	US
Kind code	B2
Filing date	May 31, 2018
Priority date	May 31, 2017
Publication date	Nov 17, 2020
Grant date	Nov 17, 2020

Abstract

An electronic device estimates a depth map of an environment based on stereo depth images captured by depth cameras having exposure times that are offset from each other in conjunction with illuminators pulsing illumination patterns into the environment. A processor of the electronic device matches small sections of the depth images from the cameras to each other and to corresponding patches of immediately preceding depth images (e.g., a spatio-temporal image patch “cube”). The processor computes a matching cost for each spatio-temporal image patch cube by converting each spatio-temporal image patch into binary codes and defining a cost function between two stereo image patches as the difference between the binary codes. The processor minimizes the matching cost to generate a disparity map, and optimizes the disparity map by rejecting outliers using a decision tree with learned pixel offsets and refining subpixels to generate a depth map of the environment.

First claim

What is claimed is:

1. A method comprising:
projecting, at an illuminator of an electronic device, a first illumination pattern into an environment of the electronic device at a frequency and a first phase and a second illumination pattern into the environment of the electronic device at the frequency and a second phase, wherein the second illumination pattern is rotated with respect to the first illumination pattern;
dynamically adjusting at least one of the first phase and the second phase;
capturing, at a depth camera of the electronic device, a first depth image of the environment at the frequency and the first phase and a second depth image of the environment at the frequency and the second phase;
computing, at a processor of the electronic device, a matching cost for corresponding portions of the first depth image and the second depth image, wherein each of the corresponding portions comprises a plurality of pixels of the first and second depth images; and
identifying the corresponding portions of the first depth image and the second depth image based on the matching cost to generate a disparity map indicating disparities between pixels of the corresponding portions of the first depth image and the second depth image; and
estimating a depth map of the environment based on the disparity map.

2. The method of claim 1 , wherein the second illumination pattern differs from the first illumination pattern.

3. The method of claim 1 , wherein the illuminator comprises a vertical cavity surface emitting laser (VCSEL).

4. The method of claim 1 , further comprising temporally varying the first and second illumination patterns.

5. The method of claim 1 , further comprising refining the disparities in the disparity map by identifying a probability that a pixel is valid and rejecting pixels with identified probabilities of validity below a threshold to estimate the depth map.

6. The method of claim 5 , wherein identifying a probability that a pixel is valid comprises identifying that the pixel is valid based on a decision tree that sparsely samples spatially neighboring pixels of the pixel.

7. The method of claim 1 , further comprising refining the disparities in the corresponding portions based on fitting a parabola to the disparities of each pixel.

8. The method of claim 1 , wherein computing the matching cost further comprises:
computing a matching cost for corresponding portions of the first depth image, the second depth image, and one or more depth images captured immediately prior to capturing the first depth image.

9. The method of claim 1 , further comprising:
tracking, using the electronic device, locations of a hand captured using the depth camera based on the depth map.

10. The method of claim 1 , wherein dynamically adjusting comprises dynamically adjusting at least one of the first phase and the second phase such that a time when the first illumination pattern is projected into the environment and a time when the second illumination pattern is projected into the environment are non-overlapping.

11. A method comprising:
pulsing, at a first illuminator of an electronic device, a first illumination pattern into an environment of the electronic device at a first phase and a first frequency;
pulsing, at a second illuminator of the electronic device, a second illumination pattern into the environment of the electronic device at a second phase and the first frequency, wherein the second illumination pattern is rotated with respect to the first illumination pattern;
dynamically adjusting at least one of the first phase and the second phase;
capturing, at a first depth camera of the electronic device, a first series of depth images of the environment at the first phase and the first frequency;
capturing, at a second depth camera of the electronic device, a second series of depth images of the environment at the second phase and the first frequency;
comparing, at a processor of the electronic device, a first patch of a first depth image of the first series of depth images to a second patch of a second depth image of the second series of depth images, wherein each patch comprises a plurality of pixels;
computing a cost function for the compared patches;
generating a disparity map indicating disparities between corresponding pixels of corresponding patches of the first depth image and the second depth image based on the cost function; and
refining the disparities of the disparity map to generate an estimated depth map of the environment.

12. The method of claim 11 , further comprising temporally varying the first and second illumination patterns.

13. The method of claim 11 , further comprising refining the disparities in the corresponding pixels by identifying a probability that a pixel is valid and rejecting pixels with identified probabilities of validity below a threshold.

14. The method of claim 11 , wherein identifying a probability that a pixel is invalid is based on a decision tree that sparsely samples spatially neighboring pixels of the pixel.

15. The method of claim 11 , wherein comparing further comprises:
comparing a first patch of a first depth image of the first series of depth images to a second patch of a second depth image of the second series of depth images and at least one patch of each of one or more depth images captured immediately prior to the first depth image.

16. The method of claim 11 , further comprising:
tracking, using the electronic device, locations of a hand captured using the first depth camera and the second depth camera based on the estimated depth map.

17. The method of claim 11 , wherein dynamically adjusting at least one of the first phase and the second phase comprises dynamically adjusting at least one of the first phase and the second phase such that a time when the first illumination pattern is pulsed into the environment and a time when the second illumination pattern is pulsed into the environment are non-overlapping.

18. An electronic device comprising:
a first illuminator to project a first illumination pattern into an environment at a first time;
a second illuminator to project a second illumination pattern into the environment at a second time different from the first time;
a first depth camera to capture a first depth image of the environment at the first time;
a second depth camera to capture a second depth image of the environment at the second time;
a processor to:
compute a matching cost for a spatio-temporal image patch, the spatio-temporal image patch generated from corresponding portions of the first depth image, the second depth image, and one or more depth images captured immediately prior to the first depth image, wherein each of the corresponding portions comprises a plurality of pixels of the depth images;
identify the corresponding portions of the first depth image and the second depth image that minimize the matching cost to generate a disparity map indicating disparities between pixels of the corresponding portions of the first depth image and the second depth image; and
refine the disparities of the disparity map to generate an estimated depth map of the environment.

19. The electronic device of claim 18 , wherein the second illumination pattern differs from the first illumination pattern.

20. The electronic device of claim 18 , wherein the second illumination pattern is rotated with respect to the first illumination pattern.

21. The electronic device of claim 18 , wherein the first and second illuminators

Assignees

• Google Llc

Inventors

• Kowdle Adarsh Prakash Murthy
• Tankovich Vladimir
• Tang Danhang
• Keskin Cem
• Taylor Jonathan James
• Davidson Philip L
• Izadi Shahram
• Fanello Sean Ryan
• Valentin Julien Pascal Christophe
• Rhemann Christoph
• Dou Mingsong
• Khamis Sameh
• Kim David

Classifications

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T2207/10152

  Varying illumination · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T2207/10024

  Color image · CPC title