Image processing device and image processing method

Patent metadata

Field	Value
Publication number	US-10812735-B2
Application number	US-201716340604-A
Country	US
Kind code	B2
Filing date	Oct 26, 2017
Priority date	Nov 15, 2016
Publication date	Oct 20, 2020
Grant date	Oct 20, 2020

Abstract

An image processing device and a method that enable removal of reflection component light from images captured from various directions are provided. The image processing device includes: an image input unit that acquires at least three types of images formed with polarization images of different polarizing directions or non-polarization images; an angle input unit that outputs reflection angles that are the angles between the image capturing directions of the acquired images and the plane direction corresponding to the normal direction of the reflecting surface, to an intensity transmittance and intensity reflectance calculation unit; a calculation unit that calculates the respective intensity transmittances and the respective intensity reflectances of S waves and P waves, using the reflection angles; and a reflection component separation unit that removes the reflected light component from the images acquired by the image input unit, using the respective intensity transmittances and the respective intensity reflectances of the S waves and P waves. These processes enable removal of reflected light components from captured images having various reflection angles.

First claim

The invention claimed is:

1. An image processing device comprising:
circuitry configured to:
acquire at least three kinds of images formed with one of a plurality of polarization images of different polarizing directions or a plurality of non-polarization images;
acquire a reflection angle that is an angle between an image capturing direction of an image of at least three kinds of images and a plane direction corresponding to a normal direction of a reflecting surface; and
remove a reflected light component from the image based on the at least three kinds of the images and the reflection angle, wherein the reflected light component is further removed based on respective intensity transmittances and respective intensity reflectances of S waves and P waves.

2. The image processing device according to claim 1 , wherein
the circuitry is further configured to:
calculate the respective intensity transmittances and the respective intensity reflectances of the S waves and the P waves, based on the reflection angle.

3. The image processing device according to claim 2 , wherein the circuitry is further configured to output relative refractive indexes of two regions separated by the reflecting surface as a boundary, to the respective intensity transmittances and the circuitry.

4. The image processing device according to claim 3 , wherein the circuitry is further configured to calculate the respective intensity transmittances and the respective intensity reflectances of the S waves and the P waves based on the reflection angle and the relative refractive indexes.

5. The image processing device according to claim 1 , wherein the circuitry is further configured to acquire the images captured by three or more imaging devices equipped with polarizing plates having different polarization angle settings.

6. The image processing device according to claim 1 , wherein the circuitry is further configured to:
acquire images captured by an imaging device equipped with a rotatable polarizing plate; and
acquire three or more kinds of polarization images captured in a state where a polarizing plate is set at rotation angles of at least three types.

7. The image processing device according to claim 1 , wherein the circuitry is further configured to acquire polarization images captured by a polarizer-stacked imaging element in which polarizers of polarization angle settings of at least three different types are stacked on respective pixels of an imaging element.

8. The image processing device according to claim 1 , wherein the circuitry is further configured to:
acquire detection information from a gyroscope that detects an angle of an imaging device that captures the images; and
calculate the reflection angle based on the detection information.

9. The image processing device according to claim 1 , wherein the circuitry is further configured to calculate the reflection angle based on an object model, wherein the object model has shape information about an object to be captured in the images.

10. The image processing device according to claim 1 , wherein the circuitry is further configured to:
acquire distance information about an object to be captured in the images; and
calculate the reflection angle based on the acquired distance information.

11. The image processing device according to claim 1 , wherein the circuitry is further configured to present a transmitted light component image, wherein the reflected light component is removed from the transmitted light component image.

12. The image processing device according to claim 11 , further comprising an angle operation device that is configured to adjust, through a user operation, the reflection angle, wherein the angle operation device is an operation unit that is configured to enable adjustment of reflection angle setting during observation of the transmitted light component image, such that an image with a small reflection component is presented.

13. The image processing device according to claim 1 , wherein the circuitry is further configured to:
compare luminances of corresponding pixels of a plurality of reflection removed images based on different reflection angle settings;
determine a pixel value with a lower luminance to be a pixel value from which a larger reflected light component has been removed; and
select the pixel value.

14. The image processing device according to claim 13 , wherein the circuitry is further configured to:
update the reflection angle through a prescribed step;
sequentially output updated reflection angle data;
generate a reflection removal result by removal of the reflected light component from an image based on the updated reflection angle data; and
compare the luminances of corresponding pixels of the plurality of reflection removed images.

15. The image processing device according to claim 14 , wherein the circuitry is further configured to:
calculate respective intensity transmittances and respective intensity reflectances of the S waves and the P waves based on the reflection angle; and
sequentially generate the reflection removal result by removal of the reflected light component from the image, using the respective intensity transmittances and the respective intensity reflectances of the S waves and the P waves based on the updated reflection angle data.

16. The image processing device according to claim 1 , wherein the circuitry is further configured to:
calculate an azimuth angle that is a first angle between a projection of a camera eye vector and an azimuth-corresponding reference axis, wherein the camera eye vector indicates the image capturing direction of a camera onto the reflecting surface;
acquire camera posture information that indicates a posture of the camera that captures the images;
calculate a zenith angle that is a second angle formed by the camera eye vector and a zenith-corresponding reference axis; and
calculate the zenith angle equivalent to the reflection angle based on input of the azimuth angle and the camera posture information.

17. The image processing device according to claim 16 , wherein the circuitry is further configured to:
select a partial image region from the image, wherein the partial image region is subjected to reflected light component removal; and
calculate the zenith angle corresponding to the partial image region.

18. The image processing device according to claim 16 , wherein the circuitry is further configured to:
divide a region on the image; and
calculate the zenith angle for each divided region.

19. An image processing method, comprising:
in an image processing device:
acquiring at least three kinds of images formed with one of a plurality of polarization images of different polarizing directions or a plurality of non-polarization images;
outputting a reflection angle that is an angle between an image capturing direction of an image of at least three kinds of images and a plane direction corresponding to a normal direction of a reflecting surface; and
removing a reflected light component from the image based on the at least three kinds of the images and the reflection angle, wherein the reflected light component is further removed based on respective intensity transmittances and respective intensity reflectances of S waves and P waves.

20. A non-transitory computer-readable medium having stored thereon, computer-executable instructions, which when executed by an image processing device, cause the image processing device to execute operations, the operations c

Assignees

• Sony Corp

Inventors

• Hirasawa Yasutaka
• Kondo Yuhi
• Oyaizu Hideki
• Kaizu Shun
• Kurita Teppei
• Zhuang Tuo
• Lu Ying
• Mihara Hajime

Classifications

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/81 Primary

  for suppressing or minimising disturbance in the image signal generation · CPC title

• H04N23/75 Primary

  by influencing optical camera components · CPC title

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• H04N5/217

  Electricity · mapped topic