Method and system for estimating the geometry of a scene

Patent metadata

Field	Value
Publication number	US-11250618-B2
Application number	US-202017075243-A
Country	US
Kind code	B2
Filing date	Oct 20, 2020
Priority date	Oct 24, 2019
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

A method of obtaining real world scale information for a scene includes obtaining at least one image of a plurality of objects in a scene; detecting at least some of the objects in the at least one image as corresponding to pre-determined objects; generating a 3D reconstruction of the scene based on the image content of the at least one image; determining a relative size of each object in the 3D reconstruction of the scene in at least one dimension, the relative size being defined in dimensions of the generated 3D reconstruction; where the relative size of each object is determined based on a distance between at least two points corresponding to that object as transformed into 3D space; obtaining a size probability distribution function for each object detected in the at least one image, each size probability distribution function defining a range of sizes in at least one dimension that a corresponding object is likely to possess in real world units; resealing the size probability distribution function for each detected object based on a corresponding relative size of that object in the 3D reconstruction; and estimating a geometry of the scene in real world units by combining the re-scaled probability distribution function for at least one detected object with the re-scaled probability distribution function for at least one other detected object.

First claim

The invention claimed is:

1. A method of obtaining real world scale information for a scene, the method comprising:
obtaining at least one image of a plurality of objects in the scene;
detecting at least some of the plurality of objects in the at least one image as corresponding to pre-determined objects;
generating a three-dimensional (3D) reconstruction of the scene based on image content of the at least one image;
determining a relative size of each object in the 3D reconstruction of the scene in at least one dimension, the relative size being defined in dimensions of the 3D reconstruction of the scene;
wherein the relative size of each object in the 3D reconstruction of the scene is determined based on a distance between at least two points corresponding to that object as transformed into 3D space;
obtaining a size probability distribution function for each object detected in the at least one image, each size probability distribution function defining a range of sizes in at least one dimension that a corresponding object is likely to possess in real world units;
rescaling the size probability distribution function for each detected object based on a corresponding relative size of that detected object in the 3D reconstruction of the scene;
and estimating a geometry of the scene in real world units by combining a re-scaled probability distribution function for at least one detected object with a re-scaled probability distribution function for at least one other detected object.

2. The method according to claim 1 , comprising:
obtaining at least two images of the scene, each image being captured from a different respective viewpoint;
detecting, for each detected object, a plurality of points in the at least one image corresponding to points on a surface of the detected objects;
determining a transformation for generating the 3D reconstruction of the scene based on corresponding image points in the at least two images; and
generating the 3D reconstruction of the scene by projecting the detected plurality of points for each detected object into the 3D space via the determined transformation.

3. The method according to claim 2 , wherein the at least two images are captured by at least one uncalibrated camera;
wherein the method further comprises determining an essential matrix for the at least two images based on the estimated geometry of the scene in real world units, and calibrating the at least one uncalibrated camera based on the determined essential matrix.

4. The method according to claim 1 , wherein estimating the geometry of the scene in real world units comprises estimating at least one of:
i. a size of at least one object in the scene;
ii. a distance of at least one object relative to a camera or each camera that captured the at least one image; and
iii. a difference in camera pose for the at least two images.

5. The method according to claim 1 , wherein estimating the geometry of the scene comprises multiplying a re-scaled probability distribution function for a first detected object with a re-scaled probability distribution function for a second detected object and determining a maximum of the multiplied re-scaled probability distribution functions as corresponding to a scale factor for the scene, the scale factor defining a conversion between a dimension measured in units of the 3D reconstruction of the scene and a corresponding dimension measured in real-world units.

6. The method according to claim 1 , comprising determining a pose of each detected object in the 3D reconstruction of the scene;
wherein determining the relative size of each detected object comprises determining a distance between at least two points defining a respective pose of that detected object; and
wherein the size probability distribution function for each detected object corresponds to the corresponding relative size of that detected object as measured between corresponding points in real world units.

7. The method according to claim 1 , comprising generating an image of a virtual object for display as part of at least one of an augmented, virtual, and mixed reality environment;
wherein at least one of a size and position of the virtual object within the at least one of the augmented, virtual, and mixed reality environment corresponds with the estimated geometry of the scene.

8. The method according to claim 1 , wherein obtaining the size probability distribution function for each detected object comprises identifying a size probability distribution function from a plurality of pre-determined size probability distribution functions that corresponds with a pre-determined object corresponding to that detected object.

9. A non-transitory, computer readable medium having computer executable instructions stored thereon, which when executed by a computer system, cause the computer system to perform a method of obtaining real world scale information for a scene by carrying out actions, comprising:
obtaining at least one image of a plurality of objects in the scene;
detecting at least some of the plurality of objects in the at least one image as corresponding to pre-determined objects;
generating a three-dimensional (3D) reconstruction of the scene based on image content of the at least one image;
determining a relative size of each object in the 3D reconstruction of the scene in at least one dimension, the relative size being defined in dimensions of the 3D reconstruction of the scene;
wherein the relative size of each object in the 3D reconstruction of the scene is determined based on a distance between at least two points corresponding to that object as transformed into 3D space;
obtaining a size probability distribution function for each object detected in the at least one image, each size probability distribution function defining a range of sizes in at least one dimension that a corresponding object is likely to possess in real world units;
resealing the size probability distribution function for each detected object based on a corresponding relative size of that detected object in the 3D reconstruction of the scene; and
estimating a geometry of the scene in real world units by combining a re-scaled probability distribution function for at least one detected object with a re-scaled probability distribution function for at least one other detected object.

10. A system comprising:
an input unit operable to obtain at least one image of a plurality of objects in a scene;
an object detector operable to detect at least some of the plurality of objects in the at least one image as corresponding to respective pre-determined objects;
a projection unit configured to generate a three-dimensional (3D) reconstruction of the scene based on image content of the at least one image;
a relative size processor configured to determine a relative size of each object in the 3D reconstruction of the scene based on a distance between points corresponding to that object in the 3D reconstruction of the scene;
a scale processor configured to obtain a plurality of size probability distribution functions, each size probability distribution function defining a range of sizes in at least one dimension that a corresponding object is likely to possess in real world units;
wherein the scale processor is configured to obtain the plurality of size probability distribution functions based on an input received from the object detector; and
wherein the scale processor is configured to re-scale a size probability distribution function obtained for each detected object based on a corresponding relative size of that detected object in the 3D reconstruction of the scene, and determine a geometry of the scene in real world units based on

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Bradley Timothy
• Cappello Fabio

Classifications

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T2207/10012

  Stereo images · CPC title

• G06T2219/2016

  Rotation, translation, scaling · CPC title

• G06T7/60 Primary

  Analysis of geometric attributes · CPC title