Apparatus and method for performing 3D estimation based on locally determined 3D information hypotheses

Patent metadata

Field	Value
Publication number	US-11170202-B2
Application number	US-201816206698-A
Country	US
Kind code	B2
Filing date	Nov 30, 2018
Priority date	Jun 1, 2016
Publication date	Nov 9, 2021
Grant date	Nov 9, 2021

Abstract

An apparatus for performing 3D estimation on the basis of pictures of at least two different views includes a hypotheses provider, a similarity measure calculator, and a 3D information determiner. The hypotheses provider locally determines 3D information hypotheses for positions of a current picture of a first view on the basis of a pre-estimate which associates a 3D information estimate to each position of a picture of the first view. The similarity measure calculator calculates, for each position of the current picture of the first view, a hypotheses of the respective position by measuring a similarity between a region similarity measure for each of the 3D information of the current picture of the first view at the respective position and a corresponding region of a second view at a position displaced relative to the respective position by a respective 3D information hypothesis. The 3D information determiner selects, for each position of the first view, the 3D information hypothesis of highest similarity measure.

First claim

The invention claimed is:

1. An apparatus for performing 3D estimation on the basis of pictures of at least two different views of the same scenery taken at different perspectives, the apparatus comprising:
a hypotheses provider configured for determining, for each position of positions of a current picture of a first view, 3D information hypotheses on the basis of a pre-estimated depth/disparity map of the current picture of the first view, wherein the 3D information hypotheses are potential 3D estimates, the positions being one of pixels of the current picture or groups of pixels of the current picture;
a similarity measure calculator for calculating, for each position of the current picture of the first view, a similarity, for each of the 3D information hypotheses of the respective position, between a region of the current picture of the first view at the respective position and a corresponding region of a second view of the at least two different views located at a position displaced relative to the respective position by the respective 3D information hypothesis; and
a 3D information determiner configured to select, for each position of the first view, the 3D information hypothesis of highest similarity in order to acquire an updated depth/disparity map,
wherein the hypotheses provider is configured to perform the determination of the 3D information hypotheses for each position of the positions of the current picture of the first view on the basis of the pre-estimated depth/disparity map and independent from the 3D information hypothesis of highest similarity selected for any surrounding position of the positions of the current picture of the first view,
wherein the pre-estimated depth/disparity map is obtained at a preceding iteration or at a preceding frame.

2. The apparatus according to claim 1 , wherein the hypotheses provider is configured for determining at least one 3D information hypothesis of the 3D information hypotheses for a current position of the current picture of the first view using updating the 3D information estimate of the current position of the picture of the first view of the pre-estimated depth/disparity map by a random update value, wherein the hypotheses provider is configured to determine the random update value based on a statistical distribution of update values.

3. The apparatus according to claim 2 , wherein the hypotheses provider is configured to determine the statistical distribution of update values based on previously determined update values related to the positions.

4. The apparatus according to claim 1 , wherein the hypotheses provider is configured for determining at least one 3D information hypothesis of the 3D information hypotheses for a current position of the current picture of the first view using one or more neighboring positions of the current position of the pre-estimated depth/disparity map.

5. The apparatus according to claim 1 , wherein the apparatus is configured to perform a further iteration for performing 3D estimation on the basis of the current picture, wherein the pre-estimated depth/disparity map of the further iteration is the updated depth/disparity map.

6. The apparatus according to claim 1 , wherein the hypotheses provider is configured for determining the 3D information hypotheses for a current position of the current picture for a further iteration using:
selecting a value of the updated depth/disparity map at the current position from an iteration preceding the further iteration;
randomly selecting one or more values of the updated depth/disparity map at one or more positions neighboring the current position of the current picture from an iteration preceding the further iteration; and
updating the value of the updated depth/disparity map at the current position of the current picture from an iteration preceding the further iteration or the one or more values of the updated depth/disparity map at one or more positons neighboring the current position of the current picture from an iteration preceding the further iteration using a random update value with respect to a suitable statistic.

7. The apparatus according to claim 1 , wherein the hypotheses provider is configured for randomly selecting 3D information hypotheses for a current position of the positions of the current picture of the first view using a statistical measure and for determining the 3D information hypotheses for a further position of the positions of the current picture of the first view different from the current positions using a further statistical measure different from the statistical measure.

8. The apparatus according to claim 7 , wherein the hypotheses provider is configured for randomly selecting at least one 3D information hypothesis of the 3D information hypotheses using a randomized spatial neighborhood as a statistical measure or a further statistical measure.

9. The apparatus according to claim 5 , wherein the apparatus is configured to perform an iteration of the 3D estimation on a representation of the picture of the first view downscaled with a first scaling factor and to perform the further iteration on a representation of the picture of the first view scaled with a second scaling factor, wherein the first scaling factor is greater than the second scaling factor.

10. The apparatus according to claim 1 ,
wherein the hypotheses provider is further configured for determining further 3D information hypotheses for positions of the current picture of a second view of the at least two different views on the basis of the pre-estimated depth/disparity map or a pre-estimated further depth/disparity map, which associates the 3 D information estimate or a further 3D information estimate to each position of a picture of the second view;
wherein the similarity measure calculator is further configured for calculating, for each position of the current picture of the second view, a similarity for each of the further 3D information hypotheses of the respective position between a region of the current picture of the second view at the respective position and a corresponding region of the first view of the at least two different views located at a position displaced relative to the respective position by the respective further 3D information hypothesis;
wherein the 3D information determiner is further configured to select, for each position of the second view, the further 3D information hypothesis of highest similarity in order to acquire an updated further depth/disparity map; and
wherein a consistency tester is configured to mutually test with a mutual test whether the selected 3D information hypothesis of the picture of the first view and the selected further 3D information hypothesis of the picture of the second view fulfill a consistency criterion, thus indicating a positive test result.

11. The apparatus according to claim 10 ,
wherein the consistency tester is configured to perform the mutual test:
for an orientation of respective regions of the picture of the first view comprising, for each position, the selected 3D information hypothesis and the picture of the second view comprising, for each position, the selected further 3D information hypothesis and/or
for center positions of respective regions of the picture of the first view comprising, for each position, the selected 3D information hypothesis and the picture of the second view comprising, for each position, the selected further 3D information hypothesis and
wherein the consistency tester is configured to acquire a consistency mask comprising a consistency measure for each position of the picture, wherein a high consistency measure indicates a deviation from selected 3D information hypothesis of th

Assignees

• Fraunhofer Ges Forschung

Inventors

• Waizenegger Wolfgang
• Schreer Oliver
• Feldmann Ingo
• Eisert Peter
• Kauff Peter

Classifications

• G06T7/55 Primary

  from multiple images · CPC title

• G06V20/653 Primary

  by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T7/536

  from perspective effects, e.g. by using vanishing points · CPC title

• G06T2200/04

  involving 3D image data · CPC title