Estimation device and method

Patent metadata

Field	Value
Publication number	US-10223804-B2
Application number	US-201615352722-A
Country	US
Kind code	B2
Filing date	Nov 16, 2016
Priority date	Nov 27, 2015
Publication date	Mar 5, 2019
Grant date	Mar 5, 2019

Abstract

A method includes associating a plurality of first feature points with a plurality of second feature points, the plurality of first feature points being included in a keyframe that represents a first captured image at a first time when a position and an orientation of a camera are successfully estimated, the plurality of second feature points being included in a second captured image captured at a second time, calculating vectors based on feature descriptors of the first feature points and feature descriptors of the second feature points for respective pairs of the first feature points and the second feature points, determining whether an association of the first feature points and the second feature points is satisfactory, and performing an estimation process that estimates a new position and a new orientation of the camera at the second time when the association is determined to be satisfactory.

First claim

What is claimed is:

1. A method executed by a processor, the method comprising:
acquiring a first image captured by a camera at a first time;
performing a first process that matches a group of first feature points extracted from the first image with map points having three-dimensional coordinates predefined for an object;
estimating, based on a number of map points that match to the first image by the first process, an accuracy of first position and orientation information Obtained by using a result of the first process, the first position and orientation information indicating a position and an orientation of a camera at the first time;
storing, in a memory, keyframe information including information of each of the group of first feature points when the estimated accuracy is higher than or equal to a first threshold;
acquiring a second image captured by the camera at a second time that is after the first time;
performing the first process on a group of second feature points extracted the second image and the map points;
estimating, based on another number of map points matched to the second image by the first process, an accuracy of second position and orientation information obtained by using another result of the first process, the second position and orientation information indicating a position and an orientation of the camera at the second time;
performing a second process that identifies pairs of respective of the group of second feature points with corresponding first feature points based on the keyframe information to generate a plurality of pairs where the estimated accuracy of the second position and orientation is less than the first threshold;
generating vectors for respective of the plurality of pairs;
generating new second position and orientation information by using at least a part of the plurality of pairs when a standard deviation of the lengths of the vectors is less than a second threshold; and
outputting the new second position and orientation information to a display control process that performs display control for displaying additional information on the second image based on the new second position and orientation information.

2. The method according to claim 1 , wherein
the keyframe information further includes the first position and orientation information, and
the new second position and orientation information is generated by further using the keyframe information when the standard deviation of the lengths of the vectors is less than the second threshold.

3. The method according to claim 1 , wherein
the first process matches one or more of the first feature points with one or more of the map points based on respective first feature descriptors of the group of first feature points and respective second feature descriptors of the second feature points,
the both of the one or more of the first feature points and the one or more of the map points have feature descriptors greater than a third threshold, and
the second process pairs respective ones of the group of second feature points with any of the first feature points in an order from one of the group of second feature points having a third feature descriptor which is most similar to any of the first feature descriptors of the group of first feature points.

4. A method executed by a processor; the method comprising:
associating a plurality of first feature points with a plurality of second feature points, the plurality of first feature points being included in a keyframe that represents a first captured image at a first time when a position of a camera and an orientation of the camera are successfully estimated, the plurality of second feature points being included in a second captured image captured at a second time by the camera;
calculating a plurality of vectors based on feature descriptors of the plurality of first feature points and feature descriptors of the plurality of second feature points for respective pairs of the plurality of first feature points and the plurality of second feature points;
determining, when a standard deviation of the lengths of the plurality of vectors is less than a first threshold, that an association of the plurality of first feature points and the plurality of second feature points is satisfactory; and
performing an estimation process that estimates a new position of the camera and a new orientation of the camera at the second time based on pairs of the plurality of first feature points and the plurality of second feature points.

5. The method according to claim 4 , wherein the estimation process at the second time is not performed when the association is determined to be not satisfactory.

6. The method according to claim 4 , further comprising:
determining, when an average of lengths of the plug of vectors is less than a second threshold, that the association is satisfactory.

7. The method according to claim 4 , further comprising:
determining, when an average of lengths of the plurality of vectors is less than a second threshold and the standard deviation of the lengths of the plurality of vectors is less than first threshold, that the association is satisfactory.

8. The method according to claim 4 , further comprising:
calculating a ratio of a number of vectors whose distance between a median and each of the plurality of vectors is greater than or equal to a third threshold, the median being determined based on of the plurality of vectors; and
determining that the association is satisfactory when the ratio is less than a fourth threshold.

9. The method according to claim 8 , wherein the estimation process estimates the new position and orientation by using pairs of first feature points and second feature points whose distance between the median and an image associated vector is less than a fifth threshold.

10. The method according to claim 5 , further comprising:
causing a display device to display information indicating that the estimation process is not performed when the association is determined to be not satisfactory.

11. A device; comprising:
a memory; and
a processor coupled to the memory and configured to:
associate a plurality of first feature points with a plurality of second feature points, the plurality of first feature points being included in a keyframe that represents a first captured image at a first time when a position of a camera and an orientation of the camera are successfully estimated, the plurality of second feature points being included in a second captured image captured at a second time by the camera;
calculate a plurality of vectors based on feature descriptors of the plurality of first feature points and feature descriptors of the plurality of second feature points for respective pairs of the plurality f first feature points and the plurality of second feature points;
determine, when a standard deviation of the lengths of the plurality of vectors is less than a first threshold, that an association of the plurality of first feature points and the plurality of second feature points is satisfactory; and
perform an estimation process that estimates a new position of the camera and a new orientation of the camera at the second time based on pairs of the plurality of first feature points and the plurality of second feature points.

12. The device according to claim 11 , wherein the estimation process at the second time is not performed when the association is determined to be not satisfactory.

13. The device according to claim 11 , wherein the processor is configured to determine, when an average of lengths of the plurality of vectors is less than a second threshold, that the assoc

Assignees

• Fujitsu Ltd

Inventors

• Moteki Atsunori
• Yamaguchi Nobuyasu
• Yoshitake Toshiyuki

Classifications

• G06T7/74 Primary

  involving reference images or patches · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title