Surface normal information producing apparatus, image capturing apparatus, surface normal information producing method, and storage medium storing surface normal information producing program

Patent metadata

Field	Value
Publication number	US-9626767-B2
Application number	US-201615001289-A
Country	US
Kind code	B2
Filing date	Jan 20, 2016
Priority date	Jan 20, 2015
Publication date	Apr 18, 2017
Grant date	Apr 18, 2017

Abstract

The surface normal information producing apparatus acquires, as captured images of an object, multiple first images in which light source positions relative to the object are mutually different and three or more second images in which polarization states are mutually different, and performs area division of a surface of the object depending on at least one of luminance information that changes corresponding to the light source positions acquired from the first images and polarization information acquired from the second images. The apparatus selects, for each divided area obtained by the area division, information to be used in production of surface normal information from information relating to change of the luminance information corresponding to the light source positions and the polarization information to produce the surface normal information.

First claim

What is claimed is:

1. A surface normal information producing apparatus configured to produce surface normal information on a normal line to a surface of an object by using a captured image acquired by image capturing of the object, the apparatus comprising:
an image acquirer configured to acquire, as the captured images, (a) multiple first images in which light source positions relative to the object are mutually different and (b) three or more second images in which polarization states are mutually different;
a divider configured to perform area division of the surface of the object depending on at least one of luminance information that changes corresponding to the light source positions acquired from the first images and polarization information acquired from the second images; and
a producer configured to select, for each divided area obtained by the area division, information to be used in production of the surface normal information from (c) information relating to change of the luminance information corresponding to the light source positions and (d) the polarization information to perform the production of the surface normal information.

2. A surface normal information producing apparatus according to claim 1 , wherein:
the producer is configured to use a reflection characteristic model when producing the surface normal information by using the information relating to the change of the luminance information corresponding to the light source positions;
the apparatus further includes a model error acquirer configured to acquire a model error amount between the luminance information and the reflection characteristic model; and
the divider is configured to perform the area division based on the model error amount.

3. A surface normal information producing apparatus according to claim 2 , wherein the model error acquirer is configured to acquire the model error amount by using a comparison result between the luminance information and a luminance value calculated using the reflection characteristic model.

4. A surface normal information producing apparatus according to claim 2 , wherein the model error acquirer is configured to acquire the model error amount by using the luminance information and a parameter of a model applied to the reflection characteristic model.

5. A surface normal information producing apparatus according to claim 2 , wherein the model error acquirer is configured to calculate the model error amount by using a variation of multiple candidates for the normal line to the surface calculated from the luminance information.

6. A surface normal information producing apparatus according to claim 2 , wherein the reflection characteristic model is Lambert's model.

7. A surface normal information producing apparatus according to claim 2 , wherein the producer is configured to produce the surface normal information by using the reflection characteristic model for, among the divided areas, at least one divided area in which the model error amount is smaller than a first model error amount.

8. A surface normal information producing apparatus according to claim 2 , wherein the producer is configured to produce the surface normal information by using the polarization information for, among the divided areas, at least one divided area in which the model error amount is larger than a second model error amount.

9. A surface normal information producing apparatus according to claim 1 , further comprising a reliability acquirer configured to acquire, when the surface normal information is produced by using the polarization information, a reliability of the production of the surface normal information by using the polarization information,
wherein the divider is configured to perform the area division based on the reliability.

10. A surface normal information producing apparatus according to claim 9 , wherein the reliability acquirer is configured to acquire the reliability by using information on a polarization degree, which is included in the polarization information.

11. A surface normal information producing apparatus according to claim 9 , wherein the reliability acquirer is configured to acquire the reliability by using a difference between a maximum value and a minimum value of a luminance of a polarized light, the values being included in the polarization information.

12. A surface normal information producing apparatus according to claim 9 , wherein the reliability acquirer is configured to acquire the reliability by using information on a polarization direction, which is included in the polarization information.

13. A surface normal information producing apparatus according to claim 9 , wherein the producer is configured to produce the surface normal information by using the polarization information for, among the divided areas, at least one divided area in which the reliability is higher than a first reliability.

14. A surface normal information producing apparatus according to claim 9 , wherein the producer is configured to produce the surface normal information by using the information relating to the change of the luminance information corresponding to the light source positions for, among the divided areas, at least one divided area in which the reliability is lower than a second reliability.

15. An image capturing apparatus comprising:
an image capturer configured to acquire a captured image by image capturing of an object; and
a surface normal information producing apparatus configured to produce surface normal information on a normal line to a surface of the object using the captured image,
wherein the surface normal information producing apparatus comprising:
an image acquirer configured to acquire, as the captured images, (a) multiple first images in which light source positions relative to the object are mutually different and (b) three or more second images in which polarization states are mutually different;
a divider configured to perform area division of the surface of the object depending on at least one of luminance information that changes corresponding to the light source positions acquired from the first images and polarization information acquired from the second images; and
a producer configured to select, for each divided area obtained by the area division, information to be used in production of the surface normal information from (c) information relating to change of the luminance information corresponding to the light source positions and (d) the polarization information to perform the production of the surface normal information.

16. An image capturing apparatus according to claim 15 , further comprising:
a light projector configured to illuminate the object; and
a mechanism capable of setting multiple light-projection positions of the light projector.

17. An image capturing apparatus according to claim 15 , wherein the image capturer is configured to acquire, as the captured images, parallax images having a parallax therebetween.

18. An image capturing apparatus according to claim 17 , wherein the apparatus is configured to acquire, as the parallax images, the multiple first images at least one viewpoint and the three or more second images at each of at least two viewpoints.

19. An image capturing apparatus according to claim 15 , wherein:
the image capturer includes an image-capturing optical system and an image sensor; and
multiple light fluxes passing through mutually different areas of a pupil of the image-capturing optical system are guided to mutually different pixels of the image

Assignees

• Canon Kk

Inventors

• Ida Yoshiaki
• Inoue Chiaki
• Kusumi Yuichi

Classifications

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T7/0073 Primary

  Physics · mapped topic

• G06T7/586 Primary

  from multiple light sources, e.g. photometric stereo · CPC title

• G01B11/24

  for measuring contours or curvatures · CPC title