Image processing device, image processing method, and image pickup device

Patent metadata

Field	Value
Publication number	US-11758290-B2
Application number	US-202017789406-A
Country	US
Kind code	B2
Filing date	Dec 18, 2020
Priority date	Jan 9, 2020
Publication date	Sep 12, 2023
Grant date	Sep 12, 2023

Abstract

The invalid pixel detection unit 32 detects an invalid pixel from a non-polarized image and a plurality of polarized images for different polarization directions obtained by performing image pickup using the polarization image pickup unit 20 . For example, the invalid pixel detection unit 32 detects a saturated pixel having a pixel value larger than a preset saturation detection threshold and a black-crushed pixel having a pixel value smaller than a preset black crushing detection threshold as invalid pixels from a non-polarized image. The polarization information generation unit 33 performs processing of generating polarization information on the basis of the non-polarized image and polarized images and switches the processing of generating the polarization information depending on the detection result of the invalid pixels in the invalid pixel detection unit 32 , so as to generate the polarization information without using the invalid pixels. Correct polarization information can be acquired.

First claim

The invention claimed is:

1. An image processing device comprising:
an invalid pixel detection unit that detects a saturated pixel and a black-crushed pixel as invalid pixels from a non-polarized image and a plurality of polarized images for different polarization directions obtained by performing image pickup using a polarization image pickup device; and
a polarization information generation unit that performs processing of generating polarization information on a basis of the non-polarized image and the polarized images, and switches processing of generating the polarization information depending on a detection result of the invalid pixels in the invalid pixel detection unit,
wherein the invalid pixel detection unit detects a pixel having a pixel value larger than a preset saturation detection threshold as the saturated pixel, and
wherein the invalid pixel detection unit and the polarization information generation unit are each implemented via at least one processor.

2. The image processing device according to claim 1 ,
wherein the polarization information generation unit generates the polarization information by using pixels of the plurality of polarized images in a case where a pixel of the non-polarized image is detected as an invalid pixel.

3. The image processing device according to claim 1 ,
wherein, in a case where a pixel of any one of the plurality of polarized images is detected as an invalid pixel, the polarization information generation unit generates the polarization information by using a pixel of the non-polarized image and a pixel of another polarized image that is not detected as an invalid pixel.

4. The image processing device according to claim 3 ,
wherein the plurality of polarized images are three or more polarized images for different polarization directions, and
the polarization information generation unit generates the polarization information by using a pixel of the non-polarized image and pixels of at least two polarized images that are not detected as invalid pixels.

5. The image processing device according to claim 4 ,
wherein the at least two polarized images are polarized images for polarization directions that are not orthogonal to each other.

6. The image processing device according to claim 1 ,
wherein the polarization information generation unit generates the polarization information by using a pixel of the non-polarized image and a pixel of a polarized image in a case where no invalid pixel is detected.

7. The image processing device according to claim 1 , wherein the invalid pixel detection unit detects the saturated pixel from the non-polarized image and the black-crushed pixel from the polarized images as the invalid pixels.

8. The image processing device according to claim 1 ,
wherein the invalid pixel detection unit detects a pixel having a pixel value smaller than a preset black crushing detection threshold as the black-crushed pixel.

9. The image processing device according to claim 8 ,
wherein the saturation detection threshold and the black crushing detection threshold are set depending on a characteristic of an image acquired by the polarization image pickup device performing the image pickup.

10. The image processing device according to claim 9 ,
wherein the black crushing detection threshold is set to be larger than an output minimum value of the polarization image pickup device by a first level, and the saturation detection threshold is set to be smaller than an output maximum value of the polarization image pickup device by a second level smaller than the first level.

11. An image processing method comprising:
detecting a saturated pixel and a black-crushed pixel as invalid pixels from a non-polarized image and a plurality of polarized images for different polarization directions obtained by performing image pickup using a polarization image pickup device;
performing processing of generating polarization information on a basis of the non-polarized image and the polarized images;
switching processing of generating the polarization information depending on a detection result of the invalid pixels; and
detecting a pixel having a pixel value larger than a preset saturation detection threshold as the saturated pixel.

12. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an image processing method, the method comprising:
detecting a saturated pixel and a black-crushed pixel as invalid pixels from a non-polarized image and a plurality of polarized images for different polarization directions obtained by performing image pickup using a polarization image pickup device;
performing processing of generating polarization information on a basis of the non-polarized image and the polarized images;
switching processing of generating the polarization information depending on a detection result of the invalid pixels; and
detecting a pixel having a pixel value larger than a preset saturation detection threshold as the saturated pixel.

Assignees

• Sony Group Corp

Inventors

• Hirasawa Yasutaka
• Kondo Yuhi
• Sun Legong
• Ono Taishi

Classifications

• H04N25/683 Primary

  by defect estimation performed on the scene signal, e.g. real time or on the fly detection · CPC title

• H04N23/843 Primary

  Demosaicing, e.g. interpolating colour pixel values · CPC title

• H04N25/13

  characterised by the spectral characteristics of the filter elements · CPC title

• H04N25/633

  by using optical black pixels · CPC title

• H04N25/77

  Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components · CPC title