Device for measuring position and orientation of imaging apparatus and method therefor

Patent metadata

Field	Value
Publication number	US-2017249752-A1
Application number	US-201715441086-A
Country	US
Kind code	A1
Filing date	Feb 23, 2017
Priority date	Feb 29, 2016
Publication date	Aug 31, 2017
Grant date	—

Abstract

An information processing device includes a holding unit which holds pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, and three-dimensional information of a feature in a space corresponding to the image in association with each other, an input unit which inputs the captured image, a derivation unit which derives a position and an orientation of the imaging apparatus when the input image is captured, an addition determination unit which determines whether a new piece of reference data is to be added, a generating unit which, if the addition determination unit determines the new piece of reference data is to be added, generates a new piece of reference data, and an updating unit which adds the generated piece of reference data and updates the pieces of reference data.

First claim

What is claimed is:

1 . An information processing device comprising:
a holding unit configured to hold pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, and three-dimensional information of a feature in a space corresponding to the image, the captured image, the position and the orientation, and the three-dimensional information being associated with each other; an input unit configured to input an image captured by the imaging apparatus; a derivation unit configured to derive a position and an orientation of the imaging apparatus when the input image is captured on the basis of the pieces of reference data and the input image; an addition determination unit configured to determine whether a new piece of reference data is to be added on the basis of the derived position and orientation and the pieces of reference data; a generating unit configured to, in a case where the addition determination unit determines the new piece of reference data is to be added, generate a new piece of reference data on the basis of the input image, the derived position and orientation, and three-dimensional information of a feature in one of the pieces of reference data; and an updating unit configured to add the generated piece of reference data and update the pieces of reference data.

2 . The information processing device according to claim 1 ,
wherein the derivation unit has a prediction unit configured to obtain prediction values for a present position and orientation of the imaging apparatus, a selection unit configured to select at least one piece of reference data from the pieces of reference data on the basis of the prediction values, and a correction unit configured to correct the prediction values by using a feature point in the selected piece of reference data to obtain the position and orientation to be derived.

3 . The information processing device according to claim 2 , wherein, in a case where a position and an orientation of the imaging apparatus in a previous frame have been obtained, the prediction unit determines the position and the orientation in the previous frame as the prediction values.

4 . The information processing device according to claim 2 , wherein, in a case where a position and an orientation of the imaging apparatus in a previous frame have not been obtained, the prediction unit performs relocalization to obtain the prediction values.

5 . The information processing device according to claim 2 , wherein the correction unit selects a predetermined or smaller number of feature points from the selected piece of reference data and corrects the prediction values by using the selected feature points.

6 . The information processing device according to claim 2 , further comprising
a distance calculation unit configured to calculate a distance between the derived position of the imaging apparatus and the position of the imaging apparatus in the selected piece of reference data, wherein the addition determination unit determines whether the new piece of reference data is to be added or not on the basis of the distance.

7 . The information processing device according to claim 2 , further comprising
a rate calculation unit configured to, in a case where feature points in the selected piece of reference data are projected on the input image, calculate a rate of feature points to be projected on the image, wherein the addition determination unit determines whether the generating unit generates a piece of reference data or not on the basis of the rate.

8 . The information processing device according to claim 2 ,
wherein the generating unit has a designation unit configured to sequentially designate feature points in the selected piece of reference data, a coordinate calculation unit configured to calculate three-dimensional coordinates and image coordinates of each of the designated feature points in a current frame in a coordinate system with reference to the imaging apparatus, a feature point determination unit configured to determine whether the designated feature point is to be added or not on the basis of the three-dimensional coordinates and the image coordinates, and an addition unit configured to add, to the pieces of reference data, the feature point determined to be added.

9 . The information processing device according to claim 2 ,
wherein the selection unit sequentially selects two or more pieces of reference data from the pieces of reference data, and wherein the correction unit sequentially corrects the prediction values by using feature points in the sequentially selected pieces of reference data to obtain the position and orientation to be derived.

10 . The information processing device according to claim 2 , wherein an attribute of the feature is included in the piece of reference data, the information processing device further comprising:
a position calculation unit configured to calculate a position of the feature on the input image on the basis of the derived position and orientation and three-dimensional information of the feature in the selected piece of reference data; and a determination unit configured to determine a degree of contribution of the feature in the piece of reference data to be generated by comparing the attribute of the feature and an attribute of the position of the feature on the input image, wherein the derivation unit derives a position and an orientation of the imaging apparatus by using the pieces of reference data updated by the updating unit and the determined degree of contribution.

11 . The information processing device according to claim 10 , wherein the attribute of the feature includes a luminance and a color.

12 . The information processing device according to claim 10 , wherein the attribute of the feature includes a distance with reference to the imaging apparatus.

13 . An information processing method for an information processing device including a holding unit configured to hold pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, and three-dimensional information of a feature in a space corresponding to the image, the captured image, the position and the orientation, and the three-dimensional information being associated with each other, the method comprising:
inputting an image captured by the imaging apparatus; deriving a position and an orientation of the imaging apparatus when the input image is captured on the basis of the pieces of reference data and the input image; determining whether a new piece of reference data is to be added on the basis of the derived position and orientation and the pieces of reference data; generating, in a case where it is determined that the new piece of reference data is to be added, a new piece of reference data on the basis of the input image, the derived position and orientation and three-dimensional information of a feature in one of the pieces of reference data; and adding the generated piece of reference data and updating the pieces of reference data.

14 . A non-transitory computer-readable storage medium storing a program causing a computer to function as:
a holding unit configured to hold pieces of reference data including a captured image, a position and an orientation of an imaging apparatus when the image is captured, and three-dimensional information of a feature in a space corresponding to the image, the captured image, the position and the orientation, and the three-dimensional information being associated wit

Assignees

• Canon Kk

Inventors

• Kotake Daisuke

Classifications

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/20004

  Adaptive image processing · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title