Image processing apparatus, image processing method, and non-transitory computer-readable medium

Patent metadata

Field	Value
Publication number	US-10059002-B2
Application number	US-201514949136-A
Country	US
Kind code	B2
Filing date	Nov 23, 2015
Priority date	Nov 28, 2014
Publication date	Aug 28, 2018
Grant date	Aug 28, 2018

Abstract

There is provided with an image processing apparatus. A captured image of a target object that is captured by an image capturing apparatus is obtained. Information that indicates a deterioration degree of the captured image is obtained for a position in the captured image. A feature of the target object is extracted from the captured image based on the deterioration degree. The feature of the target object and a feature of the three-dimensional model observed when the three-dimensional model is arranged in accordance with a predetermined position and orientation are associated. A position and orientation of the target object with respect to the image capturing apparatus are derived by correcting the predetermined position and orientation based on a result of association.

First claim

What is claimed is:

1. An image processing apparatus comprising:
one or more processors; and
at least one memory coupled to the one or more processors, the at least one memory having instructions stored thereon which, when executed by the one or more processors, cause the image processing apparatus to:
obtain a captured image of a target object that is captured by an image capturing apparatus capturing the target object, on which an illumination pattern is irradiated by an irradiation apparatus;
obtain information that indicates a deterioration degree of the captured image, for a position in the captured image;
extract a three-dimensional position of a point on an image of the target object, as a feature of the target object from the captured image based on the deterioration degree, a position of the irradiation apparatus, a position of the image capturing apparatus, and the illumination pattern;
hold a three-dimensional model of the target object;
associate the three-dimensional position of the point on the image of the target object and a three-dimensional position of a surface of the three-dimensional model that neighbors that three-dimensional position observed when the three-dimensional model is arranged in accordance with a predetermined position and orientation; and
derive a position and orientation of the target object with respect to the image capturing apparatus by correcting the predetermined position and orientation based on a result of association.

2. The image processing apparatus according to claim 1 , the at least one memory further having stored thereon instructions that, when executed by the one or more processors, cause the image processing apparatus to hold, in advance of being obtained, information that indicates an image deterioration degree for each position of a captured image that is captured by an image capturing apparatus.

3. The image processing apparatus according to claim 2 , wherein, the information that indicates the deterioration degree is held in association with a position and orientation between the target object and the image capturing apparatus.

4. The image processing apparatus according to claim 2 , the at least one memory further having stored thereon instructions that, when executed by the one or more processors, cause the image processing apparatus to hold information indicating a deterioration degree, of a captured image obtained by an image capturing apparatus by capturing the target object, at a position of each feature of an image of the target object in association with a feature that the three-dimensional model comprises that corresponds to the respective feature.

5. The image processing apparatus according to claim 1 , the at least one memory further having stored thereon instructions that, when executed by the one or more processors, cause the image processing apparatus to correct the predetermined position and orientation to make a distance between the feature of the three-dimensional model and the feature of the target object small.

6. The image processing apparatus according to claim 1 , wherein
the feature is an edge feature, and
wherein the at least one memory further has stored thereon instructions that, when executed by the one or more processors, cause the image processing apparatus to:
extract a plurality of edge features by performing edge detection processing on the captured image, and
for each of a plurality of edges that the three-dimensional model has, calculate an image position obtained by projecting the edge on a projection image based on the predetermined position and orientation, and associates an image position of edge feature of an image of the target object in the captured image with an image position of an edge of the three-dimensional model on the projection image which neighbors the edge feature of the image of the target object.

7. The image processing apparatus according to claim 1 , wherein the position and orientation of the target object with respect to the image capturing apparatus is derived by correcting the predetermined position and orientation based on the result of the associating and the deterioration degree corresponding to a position at which the feature was extracted.

8. The image processing apparatus according to claim 1 , the at least one memory further having stored thereon instructions that, when executed by the one or more processors, cause the image processing apparatus to:
set an extraction parameter used to extract the feature from the captured image in accordance with the deterioration degree, and
extract the feature from the captured image by using the extraction parameter.

9. The image processing apparatus according to claim 8 , wherein the extraction parameter is set so that a feature is extracted by filtering processing that uses a filter, wherein the higher the deterioration degree is, the larger the size of the filter is.

10. The image processing apparatus according to claim 8 , wherein the the extraction parameter is set so that a feature is extracted by filtering processing after resizing the captured image, wherein the higher the deterioration degree is, the smaller the captured image is resized.

11. The image processing apparatus according to claim 8 , the at least one memory further having stored thereon instructions that, when executed by the one or more processors, cause the image processing apparatus to:
set a plurality of extraction parameters in accordance with the deterioration degree, and
use each of the plurality of extraction parameters to extract a feature in accordance with filtering processing, and selects at least one extraction result from a plurality of extraction results in accordance with a response value of filtering processing.

12. The image processing apparatus according to claim 1 , wherein the deterioration degree indicates at least one of a blur amount and a bokeh amount of the image.

13. The image processing apparatus according to claim 1 , the at least one memory further having stored thereon instructions that, when executed by the one or more processors, cause the image processing apparatus to calculate the deterioration degree by using the three-dimensional model of the target object, based on an image capturing condition of the target object according to the image capturing apparatus.

14. The image processing apparatus according to claim 1 , the at least one memory further having stored thereon instructions that, when executed by the one or more processors, cause the image processing apparatus to calculate the deterioration degree based on a captured image obtained by the image capturing apparatus capturing the target object.

15. The image processing apparatus according to claim 1 , the at least one memory further having stored thereon instructions that, when executed by the one or more processors, cause the image processing apparatus to estimate a captured image obtained by the image capturing apparatus capturing the target object using the three-dimensional model of the target object based on an image capturing condition of the target object according to the image capturing apparatus, and calculating the deterioration degree based on the estimated image.

16. The image processing apparatus according to claim 1 , the at least one memory further having stored thereon instructions that, when executed by the one or more processors, cause the image processing apparatus to obtain the captured image by capturing the target object.

17. The image processing apparatus according to claim 1 , further comprising:
the image capturing apparatus

Assignees

• Canon Kk

Inventors

• Miyatani Sonoko
• Suzuki Tatsuya
• Fujiki Masakazu
• Kobayashi Kazuhiko
• Suzuki Masahiro
• Hirota Yuichiro
• Kotake Daisuke

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V10/443

  by matching or filtering · CPC title

• G06V10/993

  Evaluation of the quality of the acquired pattern · CPC title

• G06T5/00 Primary

  Image enhancement or restoration · CPC title

• G06T2207/30164

  Workpiece; Machine component · CPC title