Optimized camera pose estimation system

Patent metadata

Field	Value
Publication number	US-9858669-B2
Application number	US-201514921598-A
Country	US
Kind code	B2
Filing date	Oct 23, 2015
Priority date	Oct 23, 2015
Publication date	Jan 2, 2018
Grant date	Jan 2, 2018

Abstract

A camera pose estimation system is provided for estimating the position of a camera within an environment. The system may be configured to receive a 2D image captured by a camera within the environment, and interpret metadata of the 2D image to identify an estimated position of the camera. The 2D image may be registered within a 3D model of the environment, and more particularly, registered within the image plane of a synthetic camera within the model at the estimated position. A 3D point within the 3D model that has a corresponding 2D point on the 2D image may be identified. The synthetic camera and thereby the image plane and 2D image may be repositioned to a new position at which a projection line from the synthetic camera and through the corresponding 2D point intersects the corresponding 3D point, the new position being a refined position of the camera.

First claim

What is claimed is:

1. An apparatus for estimating a position of a camera within an environment, the apparatus comprising a processor and a memory storing executable instructions that, in response to execution by the processor, cause the apparatus to implement at least:
an imaging engine configured to receive a two-dimensional (2D) image captured by the camera within the environment, the 2D image having corresponding metadata with structured information indicating an estimated position of the camera within the environment;
a registration engine configured to interpret the metadata to identify the estimated position of the camera, and register the 2D image within a three-dimensional (3D) model of the environment based thereon, including the 2D image being rendered in an image plane of a synthetic camera within the 3D model at the estimated position; and
an estimation engine configured to identify a 2D point on the 2D image and a corresponding 3D point in the 3D model, and reposition the synthetic camera and thereby the image plane and 2D image to a new position of the synthetic camera at which a projection line from the synthetic camera and through the 2D point intersects the corresponding 3D point, the estimation engine being configured to output the new position of the synthetic camera as a refined position of the camera.

2. The apparatus of claim 1 , wherein a component of the estimated position of the camera has an amount of error attributable to a source of the estimated position, and the estimation engine being configured to reposition the synthetic camera includes being configured to adjust the component and thereby reduce the amount of error.

3. The apparatus of claim 1 , wherein the estimation engine being configured to identify the 2D point on the 2D image includes being configured to further identify a second 2D point on the 2D image that has a different, second corresponding 3D point in the 3D model, and
wherein the estimation engine being configured reposition the synthetic camera includes being configured to reposition the synthetic camera to the new position of the synthetic camera further at which a second projection line from the synthetic camera and through the second 21) point intersects the second corresponding 3D point.

4. The apparatus of claim 3 , wherein the estimation engine being configured to reposition the synthetic camera includes being configured to:
reposition the synthetic camera to an intermediate position at which the projection line from the synthetic camera and through the 2D point intersects the corresponding 3D point; and constrained thereby,
reposition the synthetic camera to the new position further at which the second projection line from the synthetic camera and through the second 2D point intersects the second corresponding 3D point.

5. The apparatus of claim 1 , wherein the estimation engine being configured to reposition the synthetic camera includes being configured to:
produce a projection line from the synthetic camera, through the 2D point and intersecting the 3D model at an initial 3D point; and for any distance error between the initial 3D point and the corresponding 3D point,
reposition the synthetic camera and thereby the image plane, 2D image and projection line to the new position of the synthetic camera at which the projection line from the synthetic camera and through the 2D point intersects the corresponding 3D point, and thereby reduce the distance error.

6. The apparatus of claim 3 , wherein the memory stores further executable instructions that, in response to execution by the processor, cause the apparatus to further implement at least:
a rendering engine configured to render a synthetic 2D image from the 3D model of the environment, the synthetic 2D image being rendered by the synthetic camera within the 3D model at the estimated position of the camera; and
a correlation engine configured to correlate the 2D image and the synthetic 2D image to identify a 2D point on the synthetic 2D image that is a point of correlation, the 2D point on the synthetic 2D image being output to the estimation engine that is thereby configured to identify the 2D point on the 2D image.

7. The apparatus of claim 6 , wherein the memory stores further executable instructions that, in response to execution by the processor, cause the apparatus to further implement at least:
a projection engine configured project a line from the synthetic camera and through the 2D point on the synthetic 2D image rendered in the image plane of the synthetic camera, the line intersecting the 3D model at a corresponding 3D point therein, the corresponding 3D point being output to the estimation engine that is thereby configured to identify the corresponding 3D point in the 3D model.

8. A method for estimating a position of a camera within an environment, the method comprising:
receiving a two-dimensional (2D) image captured by the camera within the environment, the 2D image having corresponding metadata with structured information indicating an estimated position of the camera within the environment;
interpreting the metadata to identify the estimated position of the camera, and registering the 2D image within a three-dimensional (3D) model of the environment based thereon, including the 2D image being rendered in an image plane of a synthetic camera within the 3D model at the estimated position; and
identifying a 2D point on the 2D image and a corresponding 3D point in the 3D model, and repositioning the synthetic camera and thereby the image plane and 2D image to a new position of the synthetic camera at which a projection line from the synthetic camera and through the 2D point intersects the corresponding 3D point, and outputting the new position of the synthetic camera as a refined position of the camera.

9. The method of claim 8 , wherein a component of the estimated position of the camera has an amount of error attributable to a source of the estimated position, and repositioning the synthetic camera includes adjusting the component and thereby reduce the amount of error.

10. The method of claim 8 , identifying the 2D point on the 2D image includes being further identifying a second 2D point on the 2D image that has a different, second corresponding 3D point in the 3D model, and
wherein repositioning the synthetic camera includes repositioning the synthetic camera to the new position of the synthetic camera further at which a second projection line from the synthetic camera and through the second 2D point intersects the second corresponding 3D point.

11. The method of claim 10 , repositioning the synthetic camera includes:
repositioning the synthetic camera to an intermediate position at which the projection line from the synthetic camera and through the 2D point intersects the corresponding 3D point; and constrained thereby,
repositioning the synthetic camera to the new position further at which the second projection line from the synthetic camera and through the second 2D point intersects the second corresponding 3D point.

12. The method of claim 8 , repositioning the synthetic camera includes:
producing a projection line from the synthetic camera, through the 2D point and intersecting the 3D model at an initial 3D point; and for any distance error between the initial 3D point and the corresponding 3D point,
repositioning the synthetic camera and thereby the image plane, 2D image and projection line to the new position of the synthetic camera at which the projection line from the synthetic camera and through the 2D point intersects the corresponding 3D point, and thereby reduce the distance error.

13. The method of clai

Assignees

• Boeing Co

Inventors

• Aughey John H

Classifications

• G06V20/17

  taken from planes or by drones · CPC title

• G06V20/13 Primary

  Satellite images · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title

• G06T7/0026

  Physics · mapped topic