Imaging device, image processing device and image processing method

Patent metadata

Field	Value
Publication number	US-10798367-B2
Application number	US-201515540396-A
Country	US
Kind code	B2
Filing date	Dec 8, 2015
Priority date	Feb 27, 2015
Publication date	Oct 6, 2020
Grant date	Oct 6, 2020

Abstract

A third imaging unit including a pixel not having a polarization characteristic is interposed between a first imaging unit and a second imaging unit including a pixel having a polarization characteristic for each of a plurality of polarization directions. A depth map is generated from a viewpoint of the first imaging unit by matching processing using a first image generated by the first imaging unit and a second image generated by the second imaging unit A normal map is generated on the basis of a polarization state of the first image. Integration processing of the depth map and the normal map is performed and a depth map with a high accuracy is generated. The depth map generated by the map integrating unit is converted into a map from a viewpoint of the third imaging unit, and an image free from deterioration can be generated.

First claim

The invention claimed is:

1. An imaging device, comprising:
a first imaging unit configured to generate a first polarized image including a first plurality of pixels, each pixel of the first plurality of pixels having a polarization characteristic for one of a plurality of polarization directions;
a second imaging unit configured to generate a second polarized image including a second plurality of pixels, each pixel of the second plurality of pixels having a polarization characteristic for one of the plurality of polarization directions; and
a third imaging unit configured to generate a non-polarized image including a third plurality of pixels, each pixel of the third plurality of pixels having no polarization characteristic,
wherein the third imaging unit is interposed between the first imaging unit and the second imaging unit,
wherein an integrated depth map is generated using the first polarized image and the second polarized image,
wherein a number of the third plurality of pixels is larger than a number of the first plurality of pixels and a number of the second plurality of pixels,
wherein the integrated depth map is converted to generate a depth value for each pixel of the third plurality of pixels, and
wherein the first imaging unit, the second imaging unit, and the third imaging unit are each implemented via at least one processor.

2. The imaging device according to claim 1 ,
wherein the first imaging unit, the second imaging unit, and the third imaging unit are matched in terms of vertical and horizontal directions, and
an optical axis position of the third imaging unit is positioned on a line connecting optical axis positions of the first imaging unit and the second imaging unit.

3. The imaging device according to claim 1 ,
wherein the first imaging unit, the second imaging unit, and the third imaging unit are included such that optical axis directions thereof are parallel each other.

4. The imaging device according to claim 1 ,
wherein the first imaging unit, the second imaging unit, and the third imaging unit are included such that entrance pupils thereof are positioned on the same line perpendicular to optical axis directions thereof.

5. The imaging device according to claim 1 ,
wherein the first imaging unit and the second imaging unit are included by being fixed on both sides of the third imaging unit.

6. The imaging device according to claim 1 , further comprising:
an adaptor attached to the third imaging unit in an attachable and detachable manner; and
a position adjusting mechanism for matching vertical and horizontal directions of the first imaging unit and the second imaging unit to those of the third imaging unit,
wherein the first imaging unit and the second imaging unit are provided to the adaptor, and
the position adjusting mechanism is provided to one of the adaptor and the third imaging unit.

7. The imaging device according to claim 6 , further comprising:
a main body unit which uses the first polarized image generated by the first imaging unit and the second polarized image generated by the second imaging unit; and
a communication unit for performing communication between the first imaging unit and the second imaging unit and the main body unit, and
wherein the main body unit and the communication unit are each implemented via at least one processor.

8. An image processing device, comprising:
a map integrating unit for generating an integrated depth map obtained by integration processing of a depth map, generated using a first polarized image generated by a first imaging unit, each pixel of the first polarized image having a polarization characteristic for one of a plurality of polarization directions and a second polarized image generated by a second imaging unit, each pixel of the second polarized image having a polarization characteristic for one of the plurality of polarization directions, and a normal map based on a polarization state of the first polarized image generated by the first imaging unit; and
a viewpoint converting unit for converting the integrated depth map generated by the map integrating unit into a map from a viewpoint of a third imaging unit interposed between the first imaging unit and the second imaging unit, the third imaging unit configured to generate a non-polarized image including a third plurality of pixels, each pixel of the third plurality of pixels having no polarization characteristic,
wherein a number of the third plurality of pixels is larger than a number of the first plurality of pixels and a number of the second plurality of pixels,
wherein the integrated depth map is converted to generate a depth value for each pixel of the third plurality of pixels, and
wherein the map integrating unit and the viewpoint converting unit are each implemented via at least one processor.

9. An image processing device, comprising:
a depth map generating unit for generating a depth map from a viewpoint of a first imaging unit by performing matching processing using a first polarized image generated by the first imaging unit configured to generate the first polarized image including a first plurality of pixels, each pixel of the first plurality of pixels having a polarization characteristic for one of a plurality of polarization directions and a second polarized image generated by a second imaging unit, each pixel of a second plurality of pixels of the second polarized image having a polarization characteristic for one of the plurality of polarization directions;
a normal map generating unit for generating a normal map on the basis of a polarization state of the first polarized image generated by the first imaging unit;
a map integrating unit for generating an integrated depth map by performing integration processing of the depth map generated by the depth map generating unit and the normal map generated by the normal map generating unit; and
a viewpoint converting unit for converting the integrated depth map generated by the map integrating unit or the depth map generated by the depth map generating unit and the normal map generated by the normal map generating unit into a map from a viewpoint of a third imaging unit interposed between the first imaging unit and the second imaging unit, the third imaging unit configured to generate a non-polarized image including a third plurality of pixels, each pixel of the third plurality of pixels having no polarization characteristic,
wherein a number of the third plurality of pixels is larger than a number of the first plurality of pixels and a number of the second plurality of pixels,
wherein the integrated depth map is converted to generate a depth value for each pixel of the third plurality of pixels, and
wherein the depth map generating unit, the normal map generating unit, the map integrating unit and the viewpoint converting unit are each implemented via at least one processor.

10. The image processing device according to claim 9 ,
wherein the viewpoint converting unit converts the depth map generated by the depth map generating unit and the normal map generated by the normal map generating unit into a map from a viewpoint of the third imaging unit.

11. The image processing device according to claim 9 ,
wherein the normal map generating unit generates the normal map on the basis of intensity of the first polarized image having three or more polarization directions.

12. The image processing device according to claim 9 ,
wherein the map integrating unit calculates a depth value not represented in the depth map from a surface shape determined on the basis of a depth value represented in the depth map and the normal map.

13

Assignees

• Sony Corp

Inventors

• Hirasawa Yasutaka
• Kondo Yuhi
• Lu Ying
• Nakatani Ayaka

Classifications

• H04N25/134

  based on three different wavelength filter elements · CPC title

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• G01B11/24

  for measuring contours or curvatures · CPC title

• G03B35/26

  using polarised or coloured light separating different viewpoint images · CPC title

• H04N13/271 Primary

  wherein the generated image signals comprise depth maps or disparity maps · CPC title