System and method for depth extraction of images with forward and backward depth prediction

Patent metadata

Field	Value
Publication number	US-9659382-B2
Application number	US-99489408-A
Country	US
Kind code	B2
Filing date	May 28, 2008
Priority date	May 28, 2008
Publication date	May 23, 2017
Grant date	May 23, 2017

Abstract

A system and method for spatiotemporal depth extraction of images with forward and backward depth prediction are provided. The system and method of the present disclosure provide for acquiring a plurality of frames, generating a first depth map of a current frame in the plurality of frames based on a depth map of a previous frame in the plurality of frames, generating a second depth map of the current frame in the plurality of frames based on a depth map of a subsequent frame in the plurality of frames, and processing the first depth map and the second depth map to produce a third depth map for the current frame.

First claim

What is claimed is:

1. In a computer, a method of matching at least two images, the method comprising:
acquiring a plurality of frames, each frame in the plurality of frames includes a first image and a second image;
generating a first depth map of a current frame in the plurality of frames based on a depth map of a previous frame in the plurality of frames;
generating a second depth map of the current frame in the plurality of frames based on a depth map of a subsequent frame in the plurality of frames; and
processing the first depth map and the second depth map to produce a third depth map for the current frame,
wherein generating first and second depth maps further include:
estimating a disparity of at least one point in the first image with at least one corresponding point in the second image, the step of estimating including performing a low-cost optimization function and computing a belief propagation function based on the results of the low-cost optimization function;
generating a disparity map based on the estimated disparity; and
converting the estimated disparity map into a depth map; and
wherein the first depth map contains an error based on the depth map of the previous frame and the second depth map contains a correction for reducing the error.

2. The method of claim 1 , wherein the estimating the disparity includes computing a pixel matching cost function.

3. The method of claim 1 , wherein the estimating the disparity includes computing a smoothness cost function.

4. The method of claim 1 , wherein the estimating the disparity includes computing a temporal cost function.

5. The method of claim 1 , wherein the first image is a left eye view and second image is a right eye view of a stereoscopic pair.

6. The method of claim 1 , wherein the second depth map contains the correction for reducing the error due to a smoothing effect accumulated during processing of the plurality of frames.

7. A system for matching at least two images, the system comprising:
means for acquiring a plurality of frames, each frame in the plurality of frames includes a first image and a second image;
means for generating a first depth map of a current frame in the plurality of frames based on a depth map of a previous frame in the plurality of frames;
means for generating a second depth map of the current frame in the plurality of frames based on a depth map of a subsequent frame in the plurality of frames; and
means for processing the first depth map and the second depth map to produce a third depth map for the current frame,
wherein the means for generating the first depth map and means for generating the second depth map further include:
means for estimating a disparity of at least one point in the first image with at least one corresponding point in the second image, the means for estimating further including means for performing a low-cost optimization function and means for computing a belief propagation function based on the results of the low-cost optimization function;
means for generating a disparity map based on the estimated disparity; and
means for converting the estimated disparity map into a depth map; and
wherein the first depth map contains an error based on the depth map of the previous frame and the second depth map contains a correction for reducing the error.

8. The system of claim 7 , wherein the means for estimating the disparity includes means for computing a pixel matching cost function.

9. The system of claim 7 , wherein the means for estimating the disparity includes means for computing a smoothness cost function.

10. The system of claim 7 , wherein the means for estimating the disparity includes computing a temporal cost function.

11. The system of claim 7 , wherein the first image is a left eye view and second image is a right eye view of a stereoscopic pair.

12. The system of claim 7 , wherein the first image is a left eye view and second image is a right eye view of a stereoscopic pair.

13. The system of claim 7 , wherein the second depth map contains the correction for reducing the error due to a smoothing effect accumulated during processing of the plurality of frames.

14. An apparatus matching at least two images, the apparatus comprising:
a memory configured for storing data and instructions;
a processor configured to acquire a plurality of frames, each frame in the plurality of frames includes a first image and a second image; generate a first depth map of a current frame in the plurality of frames based on a depth map of a previous frame in the plurality of frames; generate a second depth map of the current frame in the plurality of frames based on a depth map of a subsequent frame in the plurality of frames; and process the first depth map and the second depth map to produce a third depth map for the current frame,
wherein generating first and second depth maps further include: estimating a disparity of at least one point in the first image with at least one corresponding point in the second image, the step of estimating including performing a low-cost optimization function and computing a belief propagation function based on the results of the low-cost optimization function; generating a disparity map based on the estimated disparity; and converting the estimated disparity map into a depth map; and wherein the first depth map contains an error based on the depth map of the previous frame and the second depth map contains a correction for reducing the error.

15. The apparatus of claim 14 , wherein the processor if further configured to estimate the disparity by computing a pixel matching cost function.

16. The apparatus of claim 14 , wherein the processor if further configured to estimate the disparity by computing a smoothness cost function.

17. The apparatus of claim 14 , wherein the processor if further configured to estimate the disparity by computing a temporal cost function.

18. The apparatus of claim 14 , wherein the second depth map contains the correction for reducing the error due to a smoothing effect accumulated during processing of the plurality of frames.

Assignees

• Zhang Dong-Qing
• Izzat Izzat
• Yoon Youngshik
• Thomson Licensing

Inventors

• Zhang Dong-Qing
• Izzat Izzat
• Yoon Youngshik

Classifications

• G06T7/593 Primary

  from stereo images · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T7/579

  from motion · CPC title

• G06T2207/10012

  Stereo images · CPC title