Information processing apparatus and method of controlling the same

Patent metadata

Field	Value
Publication number	US-10957068-B2
Application number	US-201815861764-A
Country	US
Kind code	B2
Filing date	Jan 4, 2018
Priority date	Jan 6, 2017
Publication date	Mar 23, 2021
Grant date	Mar 23, 2021

Abstract

An information processing apparatus has a hold unit for holding a plurality of features of a physical space as a plurality of held features, each of the plurality of held features being related with a position in a first image captured by an image capturing apparatus at a first time, and information of an availability for a position/orientation calculation of the image capturing apparatus. The information processing apparatus associates a plurality of detected features, which are detected in a second image captured by the image capturing apparatus at a second time after the first time, with the plurality of held features, and, based on the availabilities related with the associated held features, adjusts a detected feature, among the plurality of detected features, to be used in the calculation of the position/orientation of the image capturing apparatus.

First claim

What is claimed is:

1. An information processing apparatus, comprising:
a holding unit that holds a plurality of features of a physical space as a plurality of held features,
each of the plurality of held features being related with a position in a first image captured by an image capturing apparatus at a first time, and information of an availability for a position/orientation calculation of the image capturing apparatus;
at least one memory storing a program; and
one or more processors which, by executing the program, function as:
an associating unit configured to associate a plurality of detected features, which are detected in a second image captured by the image capturing apparatus at a second time after the first time, with the plurality of held features; and
a control unit configured to, based on the availabilities related with the associated held features by the associating unit, adjust a detected feature, among the plurality of detected features, to be used in a calculation of a position/orientation of the image capturing apparatus.

2. The apparatus according to claim 1 , wherein the control unit adjusts the detected feature by selecting from the plurality of detected features a detected feature to be used in the calculation of a position/orientation of the image capturing apparatus, or amending a weight representing a contribution to the calculation of the position/orientation for each of the plurality of detected features.

3. The apparatus according to claim 2 , wherein the one or more processors further function as:
a calculating unit configured to calculate the position/orientation of the image capturing apparatus when the second image is captured by using the detected feature selected by the control unit; and
an updating unit configured to update, based on the position/orientation calculated by the calculating unit, the availability held in the holding unit.

4. The apparatus according to claim 3 , wherein the updating unit updates the positions related with the plurality of held features to positions in the second image.

5. The apparatus according to claim 3 , wherein the one or more processors further function as an obtainment unit configured to project, based on the position/orientation calculated by the calculating unit, the plurality of held features onto the second image, and obtain a reprojection error which is a distance between the position of the detected feature and a projected position of a held feature associated by the associating unit, in the second image,
wherein the updating unit decreases the availabilities corresponding to the plurality of held features based on the reprojection error.

6. The apparatus according to claim 5 , wherein the updating unit decreases the availability as the reprojection error increases.

7. The apparatus according to claim 6 , wherein the updating unit updates the availability held in the holding unit by using the smaller of, an average or weighted average of the availability held in the holding unit and the availability calculated based on the reprojection error.

8. The apparatus according to claim 1 , wherein the one or more processors further function as an updating unit configured to update the availability held in the holding unit on the basis of a result of the associating by the associating unit.

9. The apparatus according to claim 8 , wherein the holding unit further holds, in relation to each of the plurality of held features, a number of times that the associating unit consecutively succeeded at association, and
the updating unit reduces the availability of an image feature held in the holding unit as the number of times for the image feature decreases.

10. The apparatus according to claim 9 , wherein the updating unit sets the availability for a held feature for which the number of times is less than a predetermined number of times to 0.

11. The apparatus according to claim 3 , wherein the one or more processors further function as an obtainment unit configured to project, based on the position/orientation calculated by the calculating unit, the plurality of held features onto the second image, and obtain a reprojection error which is a distance between the position of the detected feature and a projection position of a held feature associated by the associating unit, in the second image,
wherein the updating unit increases the availability related to a held feature when a value for the reprojection error is consecutively smaller than a predetermined threshold value over a predetermined number of times or more.

12. The apparatus according to claim 1 , wherein
the holding unit further holds a three-dimensional map that includes three-dimensional geometric information of a feature existing in a scene,
wherein the one or more processors further function as a change unit configured to project a feature held by the three-dimensional map onto the first image, associate a held feature associated with the detected feature that is selected by the control unit with the projected feature, and change the position of the associated held feature to the projection position of the feature.

13. The apparatus according to claim 12 , wherein the one or more processors further function as an addition unit configured to add to the three-dimensional map a detected feature whose availability value is greater than a predetermined threshold value, out of the detected features associated by the associating unit.

14. The apparatus according to claim 2 , wherein the control unit selects from the plurality of detected features a detected feature whose availability value is greater than a predetermined threshold value, or increases a weight value that represents the contribution to the calculation of the position/orientation for a detected feature whose availability value is greater than the predetermined threshold value.

15. The apparatus according to claim 2 , wherein the control unit selects, out of the plurality of detected features, a predetermined number of features in an order from a feature whose availability value is highest, or increases a weight value that represents the contribution to the calculation of the position/orientation for a detected feature as the availability value of the detected feature increases.

16. The apparatus according to claim 2 , wherein the control unit separates the second image into sub-regions and, for each sub-region, selects a predetermined number of detected features in an order from a highest availability, or, for each sub-region, increases a weight value representing the contribution to the calculation of the position/orientation of the detected feature in an order from a highest availability.

17. The apparatus according to claim 2 , wherein the control unit selects the detected feature in an order from a highest availability value, so that a distance on the image becomes larger than a predetermined threshold value.

18. The apparatus according to claim 3 , wherein the one or more processors further function as:
a composition unit configured to composite, based on the position/orientation of the image capturing apparatus calculated by the calculating unit, an image of a virtual object on the first image; and
a display unit configured to display an image resulting from the compositing by the composition unit.

19. A method of controlling an information processing apparatus having a holding unit that holds a plurality of features of a physical space as a plurality of held features, each of the plurality of held features being related with a position in

Assignees

• Canon Kk

Inventors

• Tomioka Makoto
• Kotake Daisuke
• Kasuya Nozomu

Classifications

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V10/147

  Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) · CPC title

• G06V10/40

  Extraction of image or video features · CPC title

• G06V20/20

  in augmented reality scenes · CPC title