Image capturing device and imaging element, and image processing method

Patent metadata

Field	Value
Publication number	US-11108951-B2
Application number	US-201816619170-A
Country	US
Kind code	B2
Filing date	Apr 9, 2018
Priority date	Jun 13, 2017
Publication date	Aug 31, 2021
Grant date	Aug 31, 2021

Abstract

In an image capturing unit 20, an imaging element has a 4×4-pixel area in which: pixels including at least one pixel of every color component of a plurality of color components are polarization pixels of the same polarization direction; and pixels which are not the polarization pixels constitute a majority of the 4×4-pixel area, and are non-polarization pixels. The unpolarized component calculating unit 31 of the image processing unit 30 calculates unpolarized components for each pixel and for each color component by using pixel signals of polarization pixels, and pixel signals of non-polarization pixels that are generated at the image capturing unit 20. The diffuse reflection component calculating unit 32 calculates diffuse reflection components for each pixel and for each color component by using pixel signals of polarization pixels, and pixel signals of non-polarization pixels that are generated at the image capturing unit 20.

First claim

The invention claimed is:

1. An image capturing device comprising:
an image capturing unit having an imaging element with a 4×4-pixel area in which pixels including at least one pixel of every color component of a plurality of color components are polarization pixels of a same polarization direction, and pixels which are not the polarization pixels constitute a majority of the 4×4-pixel area, and are non-polarization pixels; and
an image processing unit which uses pixel signals of the polarization pixels, and pixel signals of the non-polarization pixels that are generated in the image capturing unit to calculate at least either unpolarized components or diffuse reflection components for each pixel.

2. The image capturing device according to claim 1 , wherein the image processing unit calculates the unpolarized components, and the diffuse reflection components, and calculates specular reflection components on a basis of the calculated unpolarized components and the diffuse reflection components.

3. The image capturing device according to claim 2 , wherein the image processing unit calculates, for each color component, at least either the unpolarized components or the diffuse reflection components, or the unpolarized components, the diffuse reflection components, and the specular reflection components.

4. The image capturing device according to claim 3 , wherein the image processing unit performs filtering by using the pixel signals, and calculates, for each pixel, for each color component, low frequency components, calculates, for each pixel position, high frequency component information on a basis of the pixel signals, and low frequency components of same color components as color components of the pixels, and calculates, for each color component, the unpolarized components, and the diffuse reflection components from the high frequency component information calculated for each pixel position, and the low frequency components for each color component.

5. The image capturing device according to claim 3 , wherein the image processing unit performs interpolation by using pixel signals of non-polarization pixels of same colors as colors of polarization pixels, and calculates unpolarized components of the polarization pixels, and performs interpolation by using pixel signals of polarization pixels of same colors as colors of the non-polarization pixels, and calculates diffuse reflection components of the non-polarization pixels.

6. The image capturing device according to claim 2 , wherein the image processing unit adjusts a component amount of at least either the diffuse reflection components or the specular reflection components, and combines the diffuse reflection components and the specular reflection components.

7. The image capturing device according to claim 6 , wherein the image processing unit lowers a component amount of the specular reflection components, and combines the specular reflection components.

8. The image capturing device according to claim 6 , wherein the image processing unit performs adjustment of a component amount of only the diffuse reflection components.

9. The image capturing device according to claim 6 , wherein the image processing unit performs object recognition by using the unpolarized components or the diffuse reflection components, and performs adjustment of the component amount of at least either the diffuse reflection components or the specular reflection components for each recognized object area.

10. The image capturing device according to claim 1 , wherein the image capturing unit performs generation of a pixel signal by adding signals from pixels of a same polarization characteristic and color component, and
the image processing unit performs image reduction corresponding to a number of the pixels for which the addition of signals has been performed, in a direction orthogonal to an array direction of the pixels for which the addition of the signals has been performed.

11. The image capturing device according to claim 1 , wherein an object to be captured by the image capturing unit is irradiated with polarization light of a polarization direction orthogonal to a polarization direction of the polarization pixels of the imaging element.

12. The image capturing device according to claim 11 , further comprising:
an illuminating unit performing irradiation with the polarization light.

13. An imaging element having a pixel area constituted by using pixels of a plurality of color components, the pixel area being provided with a 4×4-pixel area in which the pixels including at least one pixel of every color component of the plurality of color components are polarization pixels of a same polarization direction, and pixels which are not the polarization pixels constitute a majority of the 4×4-pixel area, and are non-polarization pixels.

14. The imaging element according to claim 13 , wherein the 4×4-pixel area is provided repetitively in the pixel area.

15. The imaging element according to claim 14 , wherein the 4×4-pixel area is provided repetitively at predetermined intervals.

16. The imaging element according to claim 13 , wherein a 2×2-pixel area in the 4×4-pixel area is constituted by the polarization pixels of the same polarization direction including a pixel of every color component.

17. The imaging element according to claim 13 , wherein four 2×2-pixel areas included in the 4×4-pixel area are constituted by pixels of every color component, and one pixel in each of the 2×2-pixel areas is one of the polarization pixels.

18. The imaging element according to claim 13 , wherein each 2×2-pixel area of four 2×2-pixel areas included in the 4×4-pixel area is constituted by pixels of a same color component.

19. The imaging element according to claim 18 , wherein the polarization pixels are provided dispersedly or adjacently.

20. An image processing method comprising:
acquiring pixel signals of polarization pixels, and pixel signals of non-polarization pixels from an image capturing unit having an imaging element with a 4×4-pixel area in which pixels including at least one pixel of every color component of a plurality of color components are the polarization pixels of a same polarization direction, and pixels which are not the polarization pixels constitute a majority of the 4×4-pixel area, and are the non-polarization pixels; and
using the pixel signals of the polarization pixels, and the pixel signals of the non-polarization pixels that are acquired from the image capturing unit to calculate at least either unpolarized components or diffuse reflection components for each pixel.

Assignees

• Sony Corp

Inventors

• Kurita Teppei
• Kaizu Shun
• Hirasawa Yasutaka

Classifications

• H04N23/60 Primary

  Control of cameras or camera modules · CPC title

• H04N25/13

  characterised by the spectral characteristics of the filter elements · CPC title

• H04N25/134

  based on three different wavelength filter elements · CPC title

• H04N23/74

  by influencing the scene brightness using illuminating means · CPC title

• H04N23/80 Primary

  Camera processing pipelines; Components thereof · CPC title