Imaging device, image processing device and image processing method

Patent metadata

Field	Value
Publication number	US-11405603-B2
Application number	US-202016942323-A
Country	US
Kind code	B2
Filing date	Jul 29, 2020
Priority date	Feb 27, 2015
Publication date	Aug 2, 2022
Grant date	Aug 2, 2022

Abstract

A third imaging unit including a pixel not having a polarization characteristic is interposed between a first imaging unit and a second imaging unit including a pixel having a polarization characteristic for each of a plurality of polarization directions. A depth map is generated from a viewpoint of the first imaging unit by matching processing using a first image generated by the first imaging unit and a second image generated by the second imaging unit. A normal map is generated on the basis of a polarization state of the first image. Integration processing of the depth map and the normal map is performed and a depth map with a high accuracy is generated. The depth map generated by the map integrating unit is converted into a map from a viewpoint of the third imaging unit, and an image free from deterioration can be generated.

First claim

The invention claimed is:

1. An imaging processing device, comprising: circuitry configured to
generate a first polarized image from a first plurality of pixels having a polarization characteristic for one of a plurality of polarization directions,
generate a second polarized image from a second plurality of pixels having a polarization characteristic for one of the plurality of polarization directions,
generate a non-polarized image from a third plurality of pixels having no polarization characteristic,
generate an integrated depth map using the first polarized image and the second polarized image, and
convert the integrated depth map to generate a depth value for each pixel of the third plurality of pixels,

wherein the circuitry is further configured to
generate a depth map based on the first polarized image and the second polarized image and a normal map based on the first polarized image,
generate the integrated depth map based on the depth map and the normal map,
convert the integrated depth map to generate the depth value for each pixel of the third plurality of pixels,
perform filter processing the first polarized image and the second polarized image to generate a matching image,
perform matching processing the matching image, and
generate the depth map by the performing matching processing.

2. The imaging processing device according to claim
wherein the circuitry is further configured to
perform edge extracting processing the matching image,
perform matching processing the matching image performed edge extracting processing, and
generate the depth map by the performing matching processing.

3. The imaging processing device according to claim 1 ,
wherein the circuitry is further configured to perform up-sampling processing of the converted integrated depth map to generate the depth value for each pixel of the third plurality of pixels.

4. An image processing method comprising:
generating a first polarized image from a first plurality of pixels having a polarization characteristic for one of a plurality of polarization directions;
generating a second polarized image from a second plurality of pixels having a polarization characteristic for one of the plurality of polarization directions;
generating a non-polarized image from a third plurality of pixels having no polarization characteristic;
generating an integrated depth map using the first polarized image and the second polarized image;
converting the integrated depth map to generate a depth value for each pixel of the third plurality of pixels;
generating a depth map based on the first polarized image and the second polarized image and a normal map based on the first polarized image;
generating the integrated depth map based on the depth map and the normal map; and
converting the integrated depth map to generate the depth value for each pixel of the third plurality of pixels.

5. The imaging processing method according to claim 4 , further comprising:
performing filter processing the first polarized image and the second polarized image to generate a matching image;
performing matching processing the matching image; and
generating the depth map by the performing matching processing.

6. The imaging processing method according to claim 5 , further comprising:
performing edge extracting processing the matching image;
performing matching processing the matching image performed edge extracting processing; and
generating the depth map by the performing matching processing.

7. The imaging processing method according to claim 4 , further comprising:
performing up-sampling processing of the converted integrated depth map to generate the depth value for each pixel of the third plurality of pixels.

8. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer, causes the computer to execute a method, the method comprising:
generating a first polarized image from a first plurality of pixels having a polarization characteristic for one of a plurality of polarization directions;
generating a second polarized image from a second plurality of pixels having a polarization characteristic for one of the plurality of polarization directions;
generating a non-polarized image from a third plurality of pixels having no polarization characteristic;
generating an integrated depth map using the first polarized image and the second polarized image;
converting the integrated depth map to generate a depth value for each pixel of the third plurality of pixels;
generating a depth map based on the first polarized image and the second polarized image and a normal map based on the first polarized image;
generating the integrated depth map based on the depth map and the normal map; and
converting the integrated depth map to generate the depth value for each pixel of the third plurality of pixels.

9. The non-transitory computer-readable medium according to claim 8 , wherein the method further comprises:
performing filter processing the first polarized image and the second polarized image to generate a matching image;
performing matching processing the matching image; and
generating the depth map by the performing matching processing.

10. The non-transitory computer-readable medium according to claim 9 , wherein the method further comprises:
performing edge extracting processing the matching image;
performing matching processing the matching image performed edge extracting processing; and
generating the depth map by the performing matching processing.

11. The non-transitory computer-readable medium according to claim 8 , wherein the method further comprises:
performing up-sampling processing of the converted integrated depth map to generate the depth value for each pixel of the third plurality of pixels.

12. An image processing method comprising:
generating a first polarized image from a first plurality of pixels having a polarization characteristic for one of a plurality of polarization directions;
generating a second polarized image from a second plurality of pixels having a polarization characteristic for one of the plurality of polarization directions;
generating a non-polarized image from a third plurality of pixels having no polarization characteristic;
generating an integrated depth map using the first polarized image and the second polarized image;
converting the integrated depth map to generate a depth value for each pixel of the third plurality of pixels; and
performing up-sampling processing of the converted integrated depth map to generate the depth value for each pixel of the third plurality of pixels.

13. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer, causes the computer to execute a method, the method comprising:
generating a first polarized image from a first plurality of pixels having a polarization characteristic for one of a plurality of polarization directions;
generating a second polarized image from a second plurality of pixels having a polarization characteristic for one of the plurality of polarization directions;
generating a non-polarized image from a third plurality of pixels having no polarization characteristic;
generating an integrated depth map using the first polarized image and the second polarized image;
converting the integrated depth map to generate a depth value for each pixel of the third plurality of pixels; and
performing up-sampling processing of the converted integrated depth map to generate the depth value for each pixel of the third plurality of pixels.

Assignees

• Sony Corp

Inventors

• Hirasawa Yasutaka
• Kondo Yuhi
• Lu Ying
• Nakatani Ayaka

Classifications

• H04N25/134

  based on three different wavelength filter elements · CPC title

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• G01B11/24

  for measuring contours or curvatures · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title

• G03B35/10

  having single camera with stereoscopic-base-defining system · CPC title