Localisation and mapping

Patent metadata

Field	Value
Publication number	US-9824450-B2
Application number	US-201314417935-A
Country	US
Kind code	B2
Filing date	Jul 25, 2013
Priority date	Jul 30, 2012
Publication date	Nov 21, 2017
Grant date	Nov 21, 2017

Abstract

A method generates a three-dimensional map of a region from successive images of the region captured from different camera poses. The method detects feature points within the captured images and designates a subset of the images as a set of keyframes each having camera pose data and respective sets of measurement data representing image positions of landmark points detected as feature points in that image. The method also includes performing bundle-adjustment to generate bundle-adjusted landmark points by iteratively refining the three dimensional spatial positions of the landmarks and the camera pose data associated with at least a subset of the keyframes. And for a feature point, not corresponding to a bundle-adjusted landmark point, detected at an intervening image which is not a keyframe and present in another intervening image which is not a keyframe, the method generates a non-bundle-adjusted point corresponding to that feature point and derives a camera pose.

First claim

The invention claimed is:

1. A method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, the method comprising:
capturing, by an image processing device, successive images of the region;
detecting, by a processor, feature points within the captured images;
designating, by the processor, a subset of the captured images as a set of keyframes each having camera pose data indicative of a camera pose and respective sets of measurement data representing image positions of landmark points detected as feature points in that image;
in respect of the keyframes, the processor performing bundle-adjustment to generate bundle-adjusted landmark points by iteratively refining the three dimensional spatial positions of the landmarks and the camera pose data associated with at least a subset of the keyframes by:
(i) deriving a difference between the measurement data for landmark points in the keyframes and corresponding image positions obtained from the camera pose data and the three dimensional spatial position of that landmark point; and
(ii) adjusting one or both of the camera pose data for one or more keyframes and the three dimensional spatial position of one or more landmark points so as to reduce the detected difference;

for a feature point, not corresponding to a bundle-adjusted landmark point, detected at an intervening image which is not a keyframe and present in another intervening image which is not a keyframe, the processor generating a non-bundle-adjusted point corresponding to that feature point; and
deriving, by the processor, a camera pose in respect of an image using the non-bundle-adjusted points and the bundle adjusted landmark points;
in which the step of generating a non-bundle-adjusted point comprises selecting a subset of one or more feature points of the intervening image to become non-bundle-adjusted points, and the selecting step comprises:
detecting the number of landmark points relating to each of a plurality of different image regions of that intervening image; and
selecting a new feature point to be added as a non-bundle-adjusted point according to a weighting applicable to the image region containing that feature point, such that the weighting is lower for an image region having a higher number of landmark points.

2. A method according to claim 1 , comprising:
at a next keyframe, performing bundle adjustment with respect to one or more of the non-bundle-adjusted points so as to convert one or more of the non-bundle-adjusted points into bundle-adjusted landmark points.

3. A method according to claim 1 , comprising the step of:
applying a position tracking filter to predict an image position corresponding to a non-bundle-adjusted point from image to image.

4. A method according to claim 3 , comprising the steps of:
comparing the predicted image position of a non-bundle-adjusted point with a detected image position of an image feature point corresponding to that non-bundle-adjusted point; and
detecting a statistical quality of the non-bundle-adjusted point from the comparison.

5. A method according to claim 4 , comprising the step of:
adopting, as non-bundle-adjusted points, those feature points having at least a threshold statistical quality.

6. A method according to claim 1 , in which the selecting step comprises:
detecting an image distance between a feature point and the image position of a nearest bundle-adjusted point in the intervening image; and
selecting a feature point as a non-bundle-adjusted point which has at least a threshold image distance from a nearest a bundle-adjusted point in the intervening image.

7. A method according to claim 1 , in which the adjusting step comprises:
deriving a statistical model of the three dimensional position of the landmark points and the camera pose; and
discarding one or more landmark points which have below a threshold fit to the statistical model.

8. A method according to claim 1 , in which the refining step operates with respect to a predetermined number of keyframes.

9. A method according to claim 1 , in which the step of generating a non-bundle-adjusted point corresponding to the feature point comprises:
detecting an initial approximation of the three dimensional position of the feature point by triangulation between two images having different camera poses.

10. A method according to claim 1 , in which the camera pose data comprises data defining a three dimensional camera position and a camera orientation.

11. A non-transitory computer-readable recording medium storing computer instructions thereon, the computer instructions, when executed by a computer, causes the computer to carry out a method of generating a three-dimensional map of a region from successive images of the region captured from different camera poses, the map comprising a set of landmark points each defined by a three dimensional spatial position and image information associated with that three dimensional position, the method comprising:
capturing successive images of the region;
detecting feature points within the captured images;
designating a subset of the captured images as a set of keyframes each having camera pose data indicative of a camera pose and respective sets of measurement data representing image positions of landmark points detected as feature points in that image;
in respect of the keyframes, performing bundle-adjustment to generate bundle-adjusted landmark points by iteratively refining the three dimensional spatial positions of the landmarks and the camera pose data associated with at least a subset of the keyframes by:
(i) deriving a difference between the measurement data for landmark points in the keyframes and corresponding image positions obtained from the camera pose data and the three dimensional spatial position of that landmark point; and
(ii) adjusting one or both of the camera pose data for one or more keyframes and the three dimensional spatial position of one or more landmark points so as to reduce the detected difference;

for a feature point, not corresponding to a bundle-adjusted landmark point, detected at an intervening image which is not a keyframe and present in another intervening image which is not a keyframe, generating a non-bundle-adjusted point corresponding to that feature point; and
deriving a camera pose in respect of an image using the non-bundle-adjusted points and the bundle adjusted landmark points;
in which generating a non-bundle-adjusted point comprises selecting a subset of one or more feature points of the intervening image to become non-bundle-adjusted points, and the selecting comprises:
detecting the number of landmark points relating to each of a plurality of different image regions of that intervening image; and
selecting a new feature point to be added as a non-bundle-adjusted point according to a weighting applicable to the image region containing that feature point, such that the weighting is lower for an image region having a higher number of landmark points.

12. The non-transitory computer-readable recording medium of claim 11 , further comprising:
at a next keyframe, performing bundle adjustment with respect to one or more of the non-bundle-adjusted points so as to convert one or more of the non-bundle-adjusted points into bundle-adjusted landmark points.

13. Image processing apparatus operable to generate a three-dimensional map of a region from successive images of the region

Assignees

• Sony Interactive Entertainment Europe Ltd

Inventors

• Martini Antonio

Classifications

• G06T7/0008 Primary

  checking presence/absence · CPC title

• G06T7/337

  involving reference images or patches · CPC title

• H04N13/0207

  Electricity · mapped topic

• H04N7/18

  Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast · CPC title

• G06T7/0044 Primary

  Physics · mapped topic