Systems and methods for stereo depth estimation using global minimization and depth interpolation

Patent metadata

Field	Value
Publication number	US-10074158-B2
Application number	US-201514631681-A
Country	US
Kind code	B2
Filing date	Feb 25, 2015
Priority date	Jul 8, 2014
Publication date	Sep 11, 2018
Grant date	Sep 11, 2018

Abstract

Aspects relate to a method of generating a high-resolution image containing depth information of an object. In one aspect, the method includes downsampling a first reference image and a second reference image from a first resolution to a second resolution, wherein the first resolution is higher than the second resolution, and wherein the first reference image and the second reference image comprising a stereo image pair. The method further includes generating a depth map at the second resolution based on global minimization techniques, using the downsampled stereo image pair. The method also includes upsampling the depth map from the second resolution to the first resolution and using a guided filter to align contours of the upsampled depth map to contours of the first reference image.

First claim

What is claimed is:

1. A method of generating a high-resolution image containing depth information of an object, the method comprising:
downsampling a first reference image at a first resolution to iteratively form a number of smaller first reference images at progressively smaller resolutions;
downsampling a second reference image at the first resolution to iteratively form a number of smaller second reference images at progressively smaller resolutions, wherein the first reference image and the second reference image are a stereo image pair, and wherein the downsampling of the first and second reference images form a stereo pair of smaller images at each downsampled resolution;
generating a depth map using the smallest downsampled first reference and second reference images;
iteratively upsampling the depth map forming progressively higher resolution depth maps using a global minimization technique for a number of iterations of upsampling to form an upsampled depth map, the resolution of each iterative depth map corresponding to one of the resolutions of the downsampled first and second reference images forming a stereo pair, wherein the number of iterations of upsampling is determined based on an execution time for forming the progressively higher resolution depth maps being less than a threshold execution time, wherein the execution time for forming the progressively higher resolution depth maps is proportional to a number of pixels in the depth maps, and wherein the number of iterations is at least two;
upsampling the upsampled depth map by interpolation without using a global minimization technique to create a depth map at the first resolution; and
aligning contours of the depth map at the first resolution to edges of the first reference image.

2. The method of claim 1 , wherein iteratively upsampling the depth map using a global minimization technique comprises minimizing an energy function between the stereo pair of first and second reference images corresponding to the resolution of the depth map.

3. The method of claim 1 , wherein the first reference image and the second reference image are captured simultaneously by cameras which are a known distance from each other.

4. The method of claim 1 , wherein upsampling the upsampled depth map by interpolation is done without using global minimization.

5. The method of claim 1 , wherein the depth map is iteratively upsampled three times forming progressively higher resolution depth maps using a global minimization technique.

6. The method of claim 1 , wherein the depth map is iteratively upsampled two times forming progressively higher resolution depth maps using a global minimization technique.

7. The method of claim 1 , wherein upsampling the upsampled depth map by interpolation is iteratively performed twice to create a depth map at the first resolution.

8. The method of claim 1 , wherein upsampling the upsampled depth map by interpolation is iteratively performed once to create a depth map at the first resolution.

9. A device for generating a high-resolution image containing depth information of an object comprising:
a processor configured to:
downsample a first reference image at a first resolution to iteratively form a number of smaller first reference images at progressively smaller resolutions;
downsample a second reference image at the first resolution to iteratively form a number of smaller second reference images at progressively smaller resolutions, wherein the first reference image and the second reference image are a stereo image pair, and wherein the downsampled first and second reference images form a stereo pair of smaller images at each downsampled resolution;
generate a depth map using the smallest downsampled first reference and second reference images;
iteratively upsample the depth map forming progressively higher resolution depth maps using a global minimization technique for a number of iterations to form an upsampled depth map, the resolution of each iterative depth map corresponding to one of the resolutions of the downsampled first and second reference images forming a stereo pair, wherein the number of iterations of upsampling is determined based on an execution time for forming the progressively higher resolution depth maps being less than a threshold execution time, wherein the execution time for forming the progressively higher resolution depth maps is proportional to a number of pixels in the depth maps, and wherein the number of iterations is at least two;
upsample the upsampled depth map by interpolation without using a global minimization technique to create a depth map at the first resolution; and
align contours of the depth map at the first resolution to edges of the first reference image.

10. The device of claim 9 , wherein iteratively upsampling the depth map using a global minimization technique comprises minimizing an energy function between the stereo pair of first and second reference images corresponding to the resolution of the depth map.

11. The device of claim 9 , wherein the processor is configured to upsample the depth map by interpolation without using global minimization.

12. The device of claim 9 , wherein the processor is configured to iteratively upsample the depth map three times forming progressively higher resolution depth maps using a global minimization technique.

13. The device of claim 9 , wherein the processor is configured to iteratively upsample the depth map two times forming progressively higher resolution depth maps using a global minimization technique.

14. The device of claim 9 , wherein the processor is configured to iteratively upsample the upsampled depth map by interpolation twice to create a depth map at the first resolution.

15. A device for generating a high-resolution image containing depth information of an object comprising:
means for downsampling a first reference image at a first resolution to iteratively form a number of smaller first reference images at progressively smaller resolutions;
means for downsampling a second reference image at the first resolution to iteratively form a number of smaller second reference images at progressively smaller resolutions, wherein the first reference image and the second reference image are a stereo image pair, and wherein the downsampling of the first and second reference images form a stereo pair of smaller images at each downsampled resolution;
means for generating a depth map using the smallest downsampled first reference and second reference images;
means for iteratively upsampling the depth map forming progressively higher resolution depth maps using a global minimization technique for a number of iterations of upsampling to form an upsampled depth map, the resolution of each iterative depth map corresponding to one of the resolutions of the downsampled first and second reference images forming a stereo pair, wherein the number of iterations of upsampling is determined based on an execution time for forming the progressively higher resolution depth maps being less than a threshold execution time, wherein the execution time for forming the progressively higher resolution depth maps is proportional to a number of pixels in the depth maps, and wherein the number of iterations is at least two;
means for upsampling the upsampled depth map by interpolation without using a global minimization technique to create a depth map at the first resolution; and
means for aligning contours of the depth map at the first resolution to edges of the first reference image.

16. The device of claim 15 , wherein iteratively upsampling

Assignees

• Qualcomm Inc

Inventors

• Siddiqui Hasib Ahmed
• Atanassov Kalin Mitkov
• Verrall Stephen Michael

Classifications

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T7/593 Primary

  from stereo images · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T7/337

  involving reference images or patches · CPC title

• G06T2207/10012

  Stereo images · CPC title