Imaging processing apparatus, image-capturing apparatus, and storage medium having image processing program stored thereon

Patent metadata

Field	Value
Publication number	US-10027942-B2
Application number	US-201414482397-A
Country	US
Kind code	B2
Filing date	Sep 10, 2014
Priority date	Mar 16, 2012
Publication date	Jul 17, 2018
Grant date	Jul 17, 2018

Abstract

A technique is disclosed to obtain first image data including a first pixel value of a first pixel having a no-parallax opening corresponding to a reference direction, a second pixel value of a second pixel having a parallax opening that causes parallax in one direction and a third pixel value of a third pixel having a parallax opening that causes parallax in the opposite direction, to calculate, for a pixel position of the second pixel, a fourth pixel value by averaging the second pixel value at the pixel position and a hypothetical third pixel value and to calculate, for a pixel position of the third pixel, a fifth pixel value by averaging the third pixel value at the pixel position and a hypothetical second pixel value, and to use the first pixel value, the fourth pixel value and the fifth pixel value to produce 2D second image data.

First claim

What is claimed is:

1. An image processing method for converting a first image into a second image, the first image being input by capturing, through a single optical system, subject images from a left viewpoint, a right viewpoint, and a reference viewpoint in a reference direction simultaneously with different pixels, using an image sensor that is represented by a color system having first to n-th (n 2:1) color components and constituted by a plurality of pixels each of which has color information corresponding to a single color component, the image sensor having a pixel array structured in such a manner that pixels of a first color component include at least three types of pixels including a left parallax pixel having an aperture mask to cause parallax in a left direction with respect to the reference direction, a right parallax pixel having an aperture mask to cause parallax in a right direction with respect to the reference direction and a no-parallax pixel having an aperture mask to create the reference viewpoint, the second image being an image in which each of the pixels has color information corresponding the first to n-th color components for the reference viewpoint,
the image processing method comprising:
producing color information corresponding to the first color component for the right viewpoint, based on captured color information corresponding to the right parallax pixel of the first color component, for a position of the left parallax pixel of the first color component of the first image;
producing color information corresponding to the first color component for the reference viewpoint, based on captured color information corresponding to the no-parallax pixel of the first color component, for a position of the left parallax pixel of the first color component of the first image;
calculating an average value between two viewpoints, or between the color information corresponding to the first color component for the left viewpoint captured by the image sensor and the produced color information corresponding to the first color component for the right viewpoint, on the position of the left parallax pixel of the first color component of the first image;
producing color information corresponding to the first color component for the left viewpoint, based on captured color information corresponding to the left parallax pixel of the first color component, for a position of the right parallax pixel of the first color component of the first image;
producing color information corresponding to the first color component for the reference viewpoint, based on captured color information corresponding to the no-parallax pixel of the first color component, for the position of the right parallax pixel of the first color component of the first image;
calculating an average value between two viewpoints, or between (i) color information corresponding to the first color component for the right viewpoint captured by the image sensor and (ii) the produced color information corresponding to the first color component for the left viewpoint, on the position of the right parallax pixel of the first color component of the first image;
calculating an average value between three viewpoints, or between (i) color information corresponding to the calculated average value of the first color component between the two viewpoints corresponding to the left viewpoint and the right viewpoint and (ii) the produced color information corresponding to the first color component for the reference viewpoint, on the position of the right parallax pixel of the first color component of the first image;
producing color information corresponding to the first color component for the left viewpoint, based on captured color information corresponding to the left parallax pixel of the first color component, for a position of the no-parallax pixel of the first color component of the first image;
producing color information corresponding to the first color component for the right viewpoint, based on captured color information corresponding to the right parallax pixel of the first color component, for the position of the no-parallax pixel of the first color component of the first image;
calculating an average value between two viewpoints, or between (i) the produced color information corresponding to the first color component for the left viewpoint and (ii) the produced color information corresponding to the first color component for the right viewpoint, on the position of the no-parallax pixel of the first color component of the first image;
calculating an average value between three viewpoints, or between (i) color information corresponding to the calculated average value of the first color component between the two viewpoints corresponding to the left viewpoint and the right viewpoint and (ii) color information corresponding to the first color component for the reference viewpoint captured by the image sensor, on the position of the no-parallax pixel of the first color component of the first image;
combining the color information corresponding to the first color component of the calculated average values between the three viewpoints with the color information corresponding to the remaining color components of the first image to output a third image in which each pixel has color information corresponding to a single color component; and
producing the second image based on the color information of the third image;
wherein the average value between the three viewpoints is calculated with the three viewpoints being allocated unequal weights.

2. The image processing method as set forth in claim 1 , further comprising: producing an image of the left viewpoint of the first color component for each pixel based on the color information corresponding to the left parallax pixel of the first color component; producing an image of the right viewpoint of the first color component for each pixel based on the color information corresponding to the right parallax pixel of the first color component; newly producing an image of the left viewpoint of the first to the n-th color components based on at least the produced image of the left viewpoint of the first color component and the second image of the reference viewpoint of the first to the n-th color components; and newly producing an image of the right viewpoint of the first to the n-th color components based on at least the produced image of the right viewpoint of the first color component and the second image of the reference viewpoint of the first to n-th color components.

3. The image processing method as set forth in claim 1 , wherein when the aperture mask of the no-parallax pixel of the image sensor has an opening area equal to a sum of opening areas of the aperture mask of the left parallax pixel and the aperture mask of the right parallax pixel, the average value between the two viewpoints, or the left and right viewpoints is determined by calculating an arithmetic average.

4. The image processing method as set forth in claim 1 , wherein when the aperture mask of the no-parallax pixel of the image sensor has the same opening area as the aperture mask of the left parallax pixel or the aperture mask of the right parallax pixel, the average value between the two viewpoints, or the left and right viewpoints is determined by calculating a geometric average.

5. The image processing method as set forth in claim 1 , wherein the average value between the three viewpoints is determined by calculating a geometric average.

6. The image processing method as set forth in claim 1 , wherein the production of the second image from the third image includes:
calculating degrees of similarity in at least two directions based on the color information of the third image;
comparing the degrees of similarity to determine a

Assignees

• Nikon Corp

Inventors

• Ishiga Kenichi
• Shibazaki Kiyoshige
• Hamashima Muneki
• Mori Susumu

Classifications

• H04N23/672

  based on the phase difference signals · CPC title

• H04N25/134

  based on three different wavelength filter elements · CPC title

• H04N23/843

  Demosaicing, e.g. interpolating colour pixel values · CPC title

• H04N25/704

  Pixels specially adapted for focusing, e.g. phase difference pixel sets · CPC title

• H04N13/207

  using a single two-dimensional [2D] image sensor · CPC title