Imaging device, image processing device and image processing method

Patent metadata

Field	Value
Publication number	US-2017366802-A1
Application number	US-201515540396-A
Country	US
Kind code	A1
Filing date	Dec 8, 2015
Priority date	Feb 27, 2015
Publication date	Dec 21, 2017
Grant date	—

Abstract

A third imaging unit including a pixel not having a polarization characteristic is interposed between a first imaging unit and a second imaging unit including a pixel having a polarization characteristic for each of a plurality of polarization directions. A depth map is generated from a viewpoint of the first imaging unit by matching processing using a first image generated by the first imaging unit and a second image generated by the second imaging unit A normal map is generated on the basis of a polarization state of the first image. Integration processing of the depth map and the normal map is performed and a depth map with a high accuracy is generated. The depth map generated by the map integrating unit is converted into a map from a viewpoint of the third imaging unit, and an image free from deterioration can be generated.

First claim

1 . An imaging device, comprising:
a first imaging unit comprising a pixel having a polarization characteristic for each of a plurality of polarization directions; a second imaging unit comprising a pixel having a polarization characteristic for each of a plurality of polarization directions; and a third imaging unit including a pixel not having a polarization characteristic and interposed between the first imaging unit and the second imaging unit.

2 . The imaging device according to claim 1 ,
wherein the first imaging unit, the second imaging unit, and the third imaging unit are matched in terms of vertical and horizontal directions, and an optical axis position of the third imaging unit is positioned on a line connecting optical axis positions of the first imaging unit and the second imaging unit.

3 . The imaging device according to claim 1 ,
wherein the first imaging unit, the second imaging unit, and the third imaging unit are included such that optical axis directions thereof are parallel each other.

4 . The imaging device according to claim 1 ,
wherein the first imaging unit, the second imaging unit, and the third imaging unit are included such that entrance pupils thereof are positioned on the same line perpendicular to optical axis directions thereof.

5 . The imaging device according to claim 1 ,
wherein the third imaging unit has a larger number of pixels than that of the first imaging unit and the second imaging unit.

6 . The imaging device according to claim 1 ,
wherein the first imaging unit and the second imaging unit are included by being fixed on both sides of the third imaging unit.

7 . The imaging device according to claim 1 , further comprising:
an adaptor attached to the third imaging unit in an attachable and detachable manner; and a position adjusting mechanism for matching vertical and horizontal directions of the first imaging unit and the second imaging unit to those of the third imaging unit, wherein the first imaging unit and the second imaging unit are provided to the adaptor, and the position adjusting mechanism is provided to one of the adaptor and the third imaging unit.

8 . The imaging device according to claim 7 , further comprising:
a main body unit which uses an image generated by the first imaging unit and an image generated by the second imaging unit; and a communication unit for performing communication between the first imaging unit and the second imaging unit and the main body unit.

9 . An image processing device, comprising:
a map integrating unit for generating an integrated depth map obtained by integration processing of a depth map, generated using a first image generated by a first imaging unit comprising a pixel having a polarization characteristic for each of a plurality of polarization directions and a second image generated by a second imaging unit having a configuration equivalent to that of the first imaging unit, and a normal map based on a polarization state of the first image generated by the first imaging unit; and a viewpoint converting unit for converting the integrated depth map generated by the map integrating unit into a map from a viewpoint of a third imaging unit interposed between the first imaging unit and the second imaging unit, the third imaging unit including a pixel having no polarization characteristic.

10 . An image processing device, comprising:
a depth map generating unit for generating a depth map from a viewpoint of a first imaging unit by performing matching processing using a first image generated by the first imaging unit comprising a pixel having a polarization characteristic for each of a plurality of polarization directions and a second image generated by a second imaging unit having a configuration equivalent to that of the first imaging unit; a normal map generating unit for generating a normal map on the basis of a polarization state of the first image generated by the first imaging unit; a map integrating unit for generating an integrated map by performing integration processing of the depth map generated by the depth map generating unit and the normal map generated by the normal map generating unit; and a viewpoint converting unit for converting the integrated map generated by the map integrating unit or the depth map generated by the depth map generating unit and the normal map generated by the normal map generating unit into a map from a viewpoint of a third imaging unit interposed between the first imaging unit and the second imaging unit, the third imaging unit including a pixel having no polarization characteristic.

11 . The image processing device according to claim 10 ,
wherein the viewpoint converting unit converts the depth map generated by the depth map generating unit and the normal map generated by the normal map generating unit into a map from a viewpoint of the third imaging unit.

12 . The image processing device according to claim 10 ,
wherein the normal map generating unit generates the normal map on the basis of intensity of the first image having three or more polarization directions.

13 . The image processing device according to claim 10 ,
wherein the map integrating unit calculates a depth value not represented in the depth map from a surface shape determined on the basis of a depth value represented in the depth map and the normal map.

14 . The image processing device according to claim 10 , further comprising:
a map converting unit for converting the depth map into a map of a large number of pixels, wherein the third imaging unit has a larger number of pixels than those of the first imaging unit and the second imaging unit, and the map converting unit converts the depth map from a viewpoint of the third imaging unit after the integration processing into a depth map corresponding to an image generated by the third imaging unit.

15 . The image processing device according to claim 10 ,
wherein the first image and the second image are in colors, and the normal map generating unit performs separation or extraction of a reflection component from the first image and generates the normal map on the basis of a polarization state of an image after the separation or the extraction.

16 . The image processing device according to claim 10 ,
wherein the first image and the second image are in black and white.

17 . An image processing method, comprising the steps of:
generating, by a map integrating unit, an integrated depth map obtained by integration processing of a depth map, generated using a first image generated by a first imaging unit comprising a pixel having a polarization characteristic for each of a plurality of polarization directions and a second image generated by a second imaging unit having a configuration equivalent to that of the first imaging unit, and a normal map based on a polarization state of the first image generated by the first imaging unit; and converting, by a viewpoint converting unit, the generated integrated depth map into a map from a viewpoint of a third imaging unit interposed between the first imaging unit and the second imaging unit, the third imaging unit including a pixel having no polarization characteristic.

Assignees

• Sony Corp

Inventors

• Hirasawa Yasutaka
• Kondo Yuhi
• Lu Ying
• Nakatani Ayaka

Classifications

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N25/134

  based on three different wavelength filter elements · CPC title

• H04N13/0271 Primary

  Electricity · mapped topic

• H04N13/0025

  Electricity · mapped topic

• H04N13/0282

  Electricity · mapped topic