Image processing apparatus, image processing method, and image pickup device

Patent metadata

Field	Value
Publication number	US-10750136-B2
Application number	US-201716339579-A
Country	US
Kind code	B2
Filing date	Aug 28, 2017
Priority date	Oct 17, 2016
Publication date	Aug 18, 2020
Grant date	Aug 18, 2020

Abstract

An image pickup unit 20 has a configuration in which non-polarizing pixels and polarizing pixels are disposed, the polarizing pixels being provided per angle in at least two polarization directions. A demosaicing unit 50 generates a non-polarized image, and a polarization component image per polarization direction, from a captured image generated by the image pickup unit 20. A polarization information generating unit 60 generates polarization information indicating the polarization characteristics of a subject included in the captured image, from the non-polarized image and the polarization component image generated by the demosaicing unit 50. As described above, the polarization information is generated with not only the polarization component image but also the highly-sensitive non-polarized image not having a decrease in the amount of light. Therefore, accurate polarization information can be acquired compared to a case where polarization information is generated on the basis of the polarization component image.

First claim

The invention claimed is:

1. An image processing apparatus comprising:
a demosaicing unit configured to generate a non-polarized image, and a polarization component image per polarization direction, from a captured image including non-polarizing pixels and polarizing pixels per at least two polarization directions; and
a polarization information generating unit configured to generate polarization information indicating a polarization characteristic of a subject included in the captured image, from the non-polarized image and the polarization component image generated by the demosaicing unit,
wherein the polarization information generating unit calculates a polarization parameter of a polarization model expression indicating the polarization characteristic at a target pixel position, on a basis of the polarization directions, a non-polarizing pixel at the target pixel position, and a polarizing pixel for each of the polarization directions, and
wherein the demosaicing unit and the polarization information generating unit are each implemented via at least one processor.

2. The image processing apparatus according to claim 1 ,
wherein the demosaicing unit calculates, for each of the polarization directions and using a target pixel, a polarizing pixel at a pixel position where the polarizing pixel is identical in polarization direction to the target pixel in peripheral pixels of the target pixel, and non-polarizing pixels equal in a pixel position to the polarizing pixel and at a pixel position of the target pixel, a pixel value of a polarizing pixel at the pixel position of the target pixel.

3. The image processing apparatus according to claim 2 ,
wherein the demosaicing unit calculates, with a relationship between a pixel average value of non-polarizing pixels at pixel positions in the identical polarization direction and a pixel value of the non-polarizing pixel at the pixel position of the target pixel, the pixel value of the polarizing pixel at the pixel position of the target pixel relative to a pixel average value of polarizing pixels at pixel positions in the identical polarization direction.

4. The image processing apparatus according to claim 1 , further comprising:
a pixel processing unit configured to generate, with a non-polarizing pixel, a pixel at a pixel position of a polarizing pixel, to generate a non-polarized image; and
a noise removal unit configured to calculate a weight to be used for smoothing filter processing, on a basis of a non-polarizing pixel at a pixel position of a target pixel and also on a basis of a non-polarizing pixel at a pixel position of a peripheral pixel that is identical in polarization direction to the target pixel, and to further calculate, using the calculated weight, the target pixel, and a polarizing pixel at the pixel position of the peripheral pixel that is identical in polarization direction to the target pixel, a pixel value of a polarizing pixel at the pixel position of the target pixel,
wherein the demosaicing unit generates a polarization component image, with the polarizing pixel after noise removal by the noise removal unit, and
wherein the pixel processing unit and the noise removal unit are each implemented via at least one processor.

5. The image processing apparatus according to claim 1 ,
wherein the polarization information generating unit uses a pixel value of the non-polarizing pixel equal in sensitivity to the polarizing pixel, as the polarization parameter.

6. The image processing apparatus according to claim 1 , further comprising:
a normal information generating unit configured to generate normal information regarding the subject on a basis of the polarization information generated by the polarization information generating unit,
wherein the normal information generating unit is implemented via at least one processor.

7. An image processing method comprising:
generating a non-polarized image, and a polarization component image per polarization direction, from a captured image including non-polarizing pixels and polarizing pixels per at least two polarization directions;
generating polarization information indicating a polarization characteristic of a subject included in the captured image, from the non-polarized image and the generated polarization component image; and
calculating a polarization parameter of a polarization model expression indicating the polarization characteristic at a target pixel position, on a basis of the polarization directions, a non-polarizing pixel at the target pixel position, and a polarizing pixel for each of the polarization directions.

8. The image processing method according to claim 7 , further comprising:
calculating, for each of the polarization directions and using a target pixel, a polarizing pixel at a pixel position where the polarizing pixel is identical in polarization direction to the target pixel in peripheral pixels of the target pixel and non-polarizing pixels equal in a pixel position to the polarizing pixel and at a pixel position of the target pixel, a pixel value of a polarizing pixel at the pixel position of the target pixel.

9. The image processing method according to claim 8 , further comprising:
calculating, with a relationship between a pixel average value of non-polarizing pixels at pixel positions in the identical polarization direction and a pixel value of the non-polarizing pixel at the pixel position of the target pixel, the pixel value of the polarizing pixel at the pixel position of the target pixel relative to a pixel average value of polarizing pixels at pixel positions in the identical polarization direction.

10. The image processing method according to claim 7 , further comprising:
generating, with a non-polarizing pixel, a pixel at a pixel position of a polarizing pixel, to generate a non-polarized image;
calculating a weight to be used for smoothing filter processing, on a basis of a non-polarizing pixel at a pixel position of a target pixel and also on a basis of a non-polarizing pixel at a pixel position of a peripheral pixel that is identical in polarization direction to the target pixel;
calculating, using the calculated weight, the target pixel, and a polarizing pixel at the pixel position of the peripheral pixel that is identical in polarization direction to the target pixel, a pixel value of a polarizing pixel at the pixel position of the target pixel; and
generating a polarization component image, with the polarizing pixel after noise removal.

11. The image processing method according to claim 7 ,
wherein a pixel value of the non-polarizing pixel equal in sensitivity to the polarizing pixel is used as the polarization parameter.

12. The image processing method according to claim 7 , further comprising:
generating normal information regarding the subject on a basis of the generated polarization information.

Assignees

• Sony Corp

Inventors

• Kaizu Shun
• Hirasawa Yasutaka
• Kondo Yuhi
• Kurita Teppei
• Lu Ying

Classifications

• H04N25/76 Primary

  Addressed sensors, e.g. MOS or CMOS sensors · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N25/134

  based on three different wavelength filter elements · CPC title

• H04N23/843 Primary

  Demosaicing, e.g. interpolating colour pixel values · CPC title

• H04N23/72

  Combination of two or more compensation controls · CPC title