Information processing apparatus, method for controlling information processing apparatus, and storage medium

Patent metadata

Field	Value
Publication number	US-11144786-B2
Application number	US-201816174458-A
Country	US
Kind code	B2
Filing date	Oct 30, 2018
Priority date	Nov 2, 2017
Publication date	Oct 12, 2021
Grant date	Oct 12, 2021

Abstract

An information processing apparatus comprising: a holding unit configured to hold a plurality of learning models for estimating geometric information based on an input image captured by an image capturing apparatus; a selection unit configured to calculate, for each of the learning models, an evaluation value that indicates suitability of the learning model to a scene of the input image, and select a learning model from the plurality of learning models based on the evaluation values; and an estimation unit configured to estimate first geometric information using the input image and the selected learning model.

First claim

What is claimed is:

1. An information processing apparatus comprising:
one or more hardware processors; and
one or more memories which store instructions executable by the one or more hardware processors to cause the information processing apparatus to perform at least:
holding a plurality of learning models, which are learned by using a captured image captured by an image capturing apparatus and depth information of the captured image as teacher data, for estimating depth information corresponding to an input image, in association with an imaging position of the captured image used as the teacher data;
selecting a learning model suitable for a scene shown in the input image from the plurality of learning models based on an evaluation result obtained from a degree of match between any one of imaging positions held in association with the learning models and an imaging position where the input image was captured; and
estimating first depth information using the input image and the selected learning model.

2. The information processing apparatus according to claim 1 , wherein the instructions are executable by the one or more hardware processors to further cause the information processing apparatus to perform:
obtaining position and orientation of the image capturing apparatus based on the first depth information.

3. The information processing apparatus according to claim 2 , wherein the instructions are executable by the one or more hardware processors to further cause the information processing apparatus to perform:
generating display information based on at least one of the input image, the first depth information, the depth information of the captured image held, the evaluation results, or the positions and orientation.

4. The information processing apparatus according to claim 3 ,
wherein in generating the display information, the instructions are executable by the one or more hardware processors to cause the information processing apparatus to perform:

generating the display information by combining, based on the first depth information, a CG image of a virtual object with the input image.

5. The information processing apparatus according to claim 3 , wherein the instructions are executable by the one or more hardware processors to further cause the information processing apparatus to perform:
estimating, for each of the learning models, third depth information based on the input image and the learning model,
wherein in generating the display information, the instructions are executable by the one or more hardware processors to cause the information processing apparatus to perform:

generating the display information by combining, based on the third depth information, a CG image of a virtual object with the input image.

6. The information processing apparatus according to claim 3 ,
wherein in generating the display information, the instructions are executable by the one or more hardware processors to cause the information processing apparatus to perform:

generating the display information by reproducing, based on the first depth information, a three-dimensional shape of a captured scene of the input image.

7. The information processing apparatus according to claim 3 , wherein the instructions are executable by the one or more hardware processors to further cause the information processing apparatus to perform:
estimating, for each of the learning models, third depth information based on the input image and the learning model, and
generating the display information by reproducing, based on the third depth information, a three-dimensional shape of a captured scene of the input image.

8. The information processing apparatus according to claim 3 , further comprising:
a display unit configured to display the display information.

9. The information processing apparatus according to claim 1 , wherein the instructions are executable by the one or more hardware processors to further cause the information processing apparatus to perform:
estimating second depth information from an input image, by stereo measurement method,
estimating, for each of the learning models, third depth information based on the input image and the learning model, and
calculating evaluation results of the learning models such that the higher a degree of match between the second depth information and the third depth information is, the higher the evaluation result is.

10. The information processing apparatus according to claim 9 , further comprising:
a sensor configured to measure a movement amount of the image capturing apparatus,
wherein the instructions are executable by the one or more hardware processors to further cause the information processing apparatus to perform:
calculating evaluation results of the learning models such that the higher the degree of match between at least one of sensor information measured by the sensor and depth information calculated based on the sensor information, and the third depth information is, the higher the evaluation result is.

11. The information processing apparatus according to claim 9 , wherein the instructions are executable by the one or more hardware processors to further cause the information processing apparatus to perform:
updating the second depth information and the third depth information, based on a second input image captured at a second time, which is different from a time at which the input image was captured by the image capturing apparatus, and
re-calculating evaluation results of the learning models such that the higher the degree of match between the updated second depth information and the updated third depth information is, the higher the evaluation result is.

12. The information processing apparatus according to claim 9 , wherein the instructions are executable by the one or more hardware processors to further cause the information processing apparatus to perform:
estimating a plurality of pieces of second depth information and a plurality of pieces of third depth information, based on a plurality of input images captured at a plurality of points in time by the image capturing apparatus, and
calculating evaluation values of the learning models such that the higher the degree of match between the plurality of pieces of second depth information and the plurality of pieces of third depth information is, the higher the evaluation result is.

13. The information processing apparatus according to claim 1 , wherein the instructions are executable by the one or more hardware processors to further cause the information processing apparatus to perform:
estimating, for each of the learning models, third depth information based on the input image and the learning model, and
calculating evaluation results of the learning models such that the more a size or shape of a known object detected from the input image conforms to the third depth information, the higher the evaluation result is.

14. The information processing apparatus according to claim 1 , wherein the instructions are executable by the one or more hardware processors to further cause the information processing apparatus to perform:
inputting a third input image and fourth depth information obtained by a second image capturing apparatus;
classifying the third input image and the fourth depth information according to a type of scene of the third input image or the fourth depth information;
holding, based on results of the classifying, the third input image and the fourth depth information according to the type of scene; and
generating the plurality of learning models for the respective types of sc

Assignees

• Canon Kk

Inventors

• Tomioka Makoto
• Kotake Daisuke
• Kasuya Nozomu
• Suzuki Masahiro

Classifications

• G06V10/765

  using rules for classification or partitioning the feature space · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/42 Primary

  Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation · CPC title

• G06F18/24765

  Rule-based classification · CPC title

• G06F18/24

  Classification techniques · CPC title