Accurate image alignment to a 3D model

Patent metadata

Field	Value
Publication number	US-9799139-B2
Application number	US-201314030714-A
Country	US
Kind code	B2
Filing date	Sep 18, 2013
Priority date	Mar 18, 2011
Publication date	Oct 24, 2017
Grant date	Oct 24, 2017

Abstract

Methods and devices for acquiring, and aligning, an image with a textured three-dimensional (3D) model are described. In one implementation, aligning can include rendering an image in the 3D model at an estimated pose and then aligning the acquired image with the rendered image. In another implementation, aligning can include projecting the acquired image onto the 3D model at an estimated pose and then aligning the projected acquired image with the textured 3D model. In still another embodiment, aligning can include projecting the acquired image onto the textured 3D model, rendering a first image in the original 3D model and a second image in the 3D model onto which the acquired image has been projected, and then aligning the first and second images.

First claim

The invention claimed is:

1. A method for positioning a two-dimensional (2D) image using a textured three dimensional (3D) model of a region of space, comprising:
obtaining a textured 3D model that represents a region of space;
acquiring a 2D image that represents at least a portion of the region of space;
determining, with respect to the textured 3D model and the acquired 2D image, an initial pose for the acquired 2D image;
synthesizing a second 2D image from the textured 3D model based on the initial pose;
aligning the acquired 2D image with the second 2D image to obtain a new estimated pose, wherein the aligning comprises:
distributing a first set of points in the acquired 2D image using image coordinates,
distributing a second set of points in the second 2D image using the same image coordinates,
selecting an area surrounding each point in the first set of points,
selecting an area surrounding each point in the second set of points, and
adjusting a position of one or more of the points in the first set of points or the second set of points based on the selected area surrounding the points of the first set of points and the second set of points; and

positioning the acquired 2D image with respect to the textured 3D model based on the new estimated pose.

2. The method of claim 1 , wherein synthesizing the second 2D image comprises rendering the second 2D image from the textured 3D model based, at least in part, on the initial pose.

3. The method of claim 1 , wherein synthesizing the second 2D image comprises:
projecting the acquired 2D image onto the textured 3D model; and
rendering the second 2D image based, at least in part, on the projection.

4. The method of claim 1 , wherein synthesizing the second 2D image comprises:
projecting the acquired 2D image onto the textured 3D model; and
obtaining a texture of a surface in the textured 3D model based, at least in part, on the projection, wherein the surface texture comprises the second 2D image.

5. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
obtain a textured three-dimensional (3D) model that represents a region of space;
acquire a two-dimensional (2D) image that represents at least a portion of the region of space;
determine, with respect to the textured 3D model and the acquired 2D image, an initial pose for the acquired 2D image;
synthesize a second 2D image from the textured 3D model based on the initial pose;
align, based on a comparison between the acquired 2D image and the second 2D image, the acquired 2D image with the second 2D image to obtain a new estimated pose; wherein the aligning comprises:
distributing a first set of points in the acquired 2D image using image coordinates,
distributing a second set of points in the second 2D image using the same image coordinates,
select an area surrounding each point in the first set of points,
select an area surrounding each point in the second set of points, and
adjust a position of one or more of the points in the first set of points or the second set of points based on the selected area surrounding the points of the first set of points and the second set of points; and

position the acquired 2D image with respect to the textured 3D model based on the new estimated pose.

6. The non-transitory program storage device of claim 5 , wherein the instructions to cause the one or more processors to acquire the 2D image comprise instructions to cause the one or more processors to acquire a street-level 2D image of a portion of the region of space.

7. The non-transitory program storage device of claim 5 , wherein the instructions to cause the one or more processors to synthesize the second 2D image comprise instructions to cause the one or more processors to render the second 2D image from the textured 3D model based, at least in part, on the initial pose.

8. The non-transitory program storage device of claim 5 , wherein the instructions to cause the one or more processors to synthesize the second 2D image comprise instructions to cause the one or more processors to:
project the acquired 2D image onto the textured 3D model; and
render the second 2D image based, at least in part, on the projection.

9. The non-transitory program storage device of claim 5 , wherein the instructions to cause the one or more processors to synthesize the second 2D image comprise instructions to cause the one or more processors to:
project the acquired 2D image onto the textured 3D model; and
obtain a texture of a surface in the textured 3D model based, at least in part, on the projection, wherein the surface texture comprises the second 2D image.

10. The non-transitory program storage device of claim 5 , wherein the instructions to cause the one or more processors to synthesize the second 2D image comprise instructions to cause the one or more processors to obtain a plurality of second 2D images from around a portion of the textured 3D model corresponding to the initial pose.

11. The non-transitory program storage device of claim 10 , wherein the instructions to cause the one or more processors to align comprise instructions to cause the one or more processors to:
determine a displacement between the acquired 2D image and each of the plurality of second 2D images;
select an image from the plurality of second 2D images having a smallest displacement; and
align the acquired 2D image with the selected 2D image to obtain a new estimated pose.

12. The non-transitory program storage device of claim 11 , further comprising instructions to cause the one or more processors to repeat the instructions to cause the one or more processors to determine a displacement, select an image, and align the acquired 2D image one or more additional times.

13. The non-transitory program storage device of claim 12 , wherein the instructions to cause the one or more processors to repeat are continued until a determined displacement is less than a specified value.

14. The non-transitory program storage device of claim 11 , wherein the instructions to cause the one or more processors to determine a displacement comprise instructions to cause the one or more processors to:
identify first features in the acquired 2D image;
identify second features in the second 2D image from the plurality of second 2D images; and
compare the first and second features, wherein the more similar the first and second features the lower the displacement difference between the acquired 2D image and the second 2D image.

15. The non-transitory program storage device of claim 5 , further comprising instructions to cause the one or more processors to:
extract texture and/or geometric information from the acquired 2D image; and
include the extracted information into the textured 3D model.

16. The non-transitory program storage device of claim 5 , wherein the instructions to cause the one or more processors to align the acquired 2D image with the second 2D image to obtain a new estimated pose comprise instructions to cause the one or more processors to:
generate a scale space representation of the acquired 2D image and the second 2D image based on the first set of points and the second set of points, and
correlate the acquired 2D image and the second 2D image based on the scale space representation.

17. The non-transitory program storage device of claim 16 , further comprising instructions to cause the one or more processors to:
measure a correlation between each selected area of the acquired 2D image and the corresponding selected ar

Assignees

• Apple Inc

Inventors

• Sievert Andreas Maximilian
• Kolbe Viktor

Classifications

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T17/05

  Geographic models · CPC title

• G06T7/75 Primary

  involving models · CPC title

• G06T15/04

  Texture mapping · CPC title