Hybrid recursive analysis of spatio-temporal objects

Patent metadata

Field	Value
Publication number	US-9947096-B2
Application number	US-201414287274-A
Country	US
Kind code	B2
Filing date	May 27, 2014
Priority date	Nov 30, 2011
Publication date	Apr 17, 2018
Grant date	Apr 17, 2018

Abstract

A method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives has: providing at least two hypotheses for the 3D-information; performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion; updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve the matching criterion or another criterion; and determining the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the action of updating the test-result hypothesis. A corresponding computer readable digital storage medium and a 3D-information generator are also described. Further embodiments perform a correspondence analysis between projections of spatio-temporal objects (STO) in multiple images to select a particular spatio-temporal object on the basis of said correspondence analysis.

First claim

The invention claimed is:

1. A method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives, the method comprising:
providing at least two hypotheses for the 3D-information;
performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion by selecting one of the at least two hypotheses as the test result hypothesis, wherein the multi-hypotheses test comprises determining projections of at least two 3D spatio-temporal objects defined by the at least two hypotheses on a plurality of image planes corresponding to the multiple images and performing a correspondence analysis between the projections in the multiple images to determine the test-result hypothesis;
updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve the matching criterion or another criterion, the parameter set comprising at least one of a 3D position, a 3D orientation, a deformation, and material properties;
determining the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the action of updating the test-result hypothesis; and
repeating the steps of providing, performing and updating as a subsequent iteration,
wherein the step of providing in the subsequent iteration comprises deriving at least one of the at least two hypotheses for the 3D-information from the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration, wherein the deriving comprises at least one of spatially translating, re-orienting and deforming the 3D spatio-temporal object of the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration.

2. The method according to claim 1 , wherein the action of updating the test-result hypothesis further comprises:
comparing an updated test-result hypothesis and the test-result hypothesis with respect to the particular matching criterion or a further matching criterion; and
choosing the updated test-result hypothesis or the test-result hypothesis as the resulting hypothesis in dependence on a result of the comparison.

3. The method according to claim 1 , wherein the provision of the at least two hypotheses comprises retrieving selected hypotheses relative to one or more previous recursion(s) of the method from a hypotheses memory, the one or more previous recursion(s) comprising a defined or known spatial and temporal relation to a current recursion of the method.

4. The method according to claim 3 , wherein the retrieved selected hypotheses are a result of a recursion of the one or more previous recursion(s) thereby establishing a feed-back loop that includes the step of performing and the step of updating.

5. The method according to claim 1 , wherein the update of the test-result hypothesis comprises a multi-dimensional parameter optimization of the parameter set of the test-result hypothesis.

6. The method according to claim 1 , wherein the update of the test-result hypothesis comprises a recursive determination of parameter variations on the basis of at least one or more of:
a test-result hypothesis determined during a previous recursion of the method and an updated test-result hypothesis determined during the previous recursion.

7. The method according to claim 1 , wherein the particular matching criterion is at least one of a hypothesis confidence and a probability of a given hypothesis.

8. The method according to claim 1 , wherein the projections of the at least two 3D objects are simplified versions.

9. The method according to claim 1 , wherein the at least two hypotheses define at least one of the 3D position, the 3D orientation, a material property, and a surface reflectance of at least two corresponding 3D spatio-temporal objects.

10. The method according to claim 1 ,
wherein the at least two hypotheses define at least two 3D spatio-temporal objects; and
wherein the update of the test-result hypothesis comprises updating the parameter set according to the deformation of a corresponding 3D spatio-temporal object between a previous recursion and a current recursion of the method, the previous recursion regarding a temporally previous version of the corresponding 3D spatio-temporal object.

11. The method according to claim 1 , wherein within the multi-hypotheses test a relatively small number of hypotheses is evaluated.

12. The method according to claim 1 , wherein at least one of the at least two hypotheses is a result of a recursion, the recursion including the step of performing and the step of updating, thereby establishing a feed-back loop.

13. The method according to claim 1 , wherein the deriving is performed so that at least two of the at least two hypotheses for the 3D-information are derived from the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration.

14. The method according to claim 1 , wherein the step of providing in the subsequent iteration includes deriving at least a further one of the at least two hypotheses for the 3D-information from 3D information concerning a previous video frame.

15. A non-transitory computer readable digital storage medium having stored thereon a computer program comprising a program code for performing, when running on a computer, a method for generating 3D-information from multiple images showing a 3D scene from multiple perspectives, the method comprising:
providing at least two hypotheses for the 3D-information;
performing a multi-hypotheses test by matching the at least two hypotheses to the multiple images and determining a test-result hypothesis that fulfills a particular matching criterion by selecting one of the at least two hypotheses as the test result hypothesis, wherein the multi-hypotheses test comprises determining projections of at least two 3D spatio-temporal objects defined by the at least two hypotheses on a plurality of image planes corresponding to the multiple images and performing a correspondence analysis between the projections in the multiple images to determine the test-result hypothesis;
updating the test-result hypothesis by varying a parameter set of the test-result hypothesis to further improve the matching criterion or another criterion, the parameter set comprising at least one of a 3D position, a 3D orientation, a deformation, and material properties;
determining the 3D-information on the basis of the parameter set of a resulting hypothesis provided by the action of updating the test-result hypothesis; and
repeating the steps of providing, performing and updating as a subsequent iteration,
wherein the step of providing in the subsequent iteration comprises deriving at least one of the at least two hypotheses for the 3D-information from the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration, wherein the deriving comprises at least one of spatially translating, re-orienting and deforming the 3D spatio-temporal object of the resulting hypothesis provided by the action of updating the test-result hypothesis preceding the subsequent iteration.

16. A 3D-information generator comprising:
an interface that receives multiple images showing a 3D scene from multiple perspectives;
a hypotheses provider that provides at least two hypotheses for the 3D-information;
a multi-hypotheses tester that performs a multi-hypotheses test by matching the at lea

Assignees

• Fraunhofer Ges Forschung

Inventors

• Atzpadin Nicole
• Waizenegger Wolfgang
• Schreer Oliver
• Feldmann Ingo
• Kauff Peter
• Eisert Peter

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/593 Primary

  from stereo images · CPC title

• G06T7/004 Primary

  Physics · mapped topic

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title