Pixel interpolation device and pixel interpolation method, and image processing device, and program and recording medium

Patent metadata

Field	Value
Publication number	US-11748852-B2
Application number	US-201816973041-A
Country	US
Kind code	B2
Filing date	Jun 27, 2018
Priority date	Jun 27, 2018
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

A first group of interpolation calculators performing interpolation by mean preserving interpolation calculation, and a second group of interpolation calculators performing interpolation by interpolation other than the mean preserving interpolation calculation are provided. Each of the interpolation calculators performs interpolation calculation for the missing pixel to calculate an interpolation candidate value (M(h)), and calculates test interpolation values (M(t)) treating pixels in a vicinity of the missing pixel as test pixels. A decider ( 4 ) selects one of the plurality of interpolation calculators based on results of the test interpolation. An outputter ( 5 ) selects the interpolation candidate value outputted from the interpolation calculator selected by the decider ( 4 ), among the plurality of interpolation candidate values (M(h)), and outputs the selected interpolation candidate value. Interpolation can be performed such that the interpolated pixel does not look unnatural, and yet the storage capacity and the data processing amount that are required are relatively small.

First claim

The invention claimed is:

1. A pixel interpolation device comprising processing circuitry configured as:
a plurality of interpolation calculators each interpolating a pixel value of a missing pixel in an input image, based on pixel values of pixels constituting said input image;
a decider; and
an outputter, wherein
each of said plurality of interpolation calculators interpolates the pixel value of said missing pixel, and outputs the interpolated pixel value as an interpolation candidate value, treats a plurality of non-missing pixels in a vicinity of said missing pixel as test pixels, interpolates a pixel value of each of said test pixels, and outputs the interpolated pixel value as a test interpolated value of the test pixel,
the interpolation of said test pixels is performed in a same interpolation method as the interpolation of said missing pixel,
said plurality of interpolation calculators include a first group of interpolation calculators, and a second group of interpolation calculators,
said first group of interpolation calculators perform the interpolation by mean preserving interpolation calculation,
said first group of interpolation calculators differ from each other in a number of pixels referenced,
said second group of interpolation calculators perform the interpolation by interpolation calculation different from said mean preserving interpolation calculation,
said second group of interpolation calculators differ from each other in at least one of an interpolation calculation method and a number of pixels referenced,
said decider calculates error index values pertaining to respective interpolation calculators, based on absolute difference values between said test interpolated values pertaining to the respective test pixels in the vicinity of each missing pixel, and the pixel values of the test pixels, and selects one of the plurality of interpolation calculators based on the calculated error index values, and
said outputter selects, for each missing pixel, the interpolation candidate value outputted from the interpolation calculator selected by said decider, among the interpolation candidate values outputted from the plurality of interpolation calculators, and outputs the selected interpolation candidate value.

2. The pixel interpolation device as set forth in claim 1 , wherein
said mean preserving interpolation calculation interpolates the pixel value of the missing pixel such that a mean value of pixel values of pixels in a missing part consisting of a plurality of pixels which include the missing pixel is equal to a mean value of pixel values of pixels in a non-missing part consisting of a plurality of pixels which do not include the missing pixel.

3. The pixel interpolation device as set forth in claim 2 , wherein a number of pixels in said missing part and a number of pixels in said non-missing part are equal to each other.

4. The pixel interpolation device as set forth in claim 3 , wherein, in the mean preserving interpolation calculation,
a sum of pixel values of the pixels other than the missing pixel, in said missing part is determined as a missing part total,
a sum of pixel values of the pixels in said non-missing part is determined as a non-missing part total, and
a difference value obtained by subtracting said missing part total from said non-missing part total is determined as the pixel value of said missing pixel.

5. The pixel interpolation device as set forth in claim 2 , wherein a number of pixels in said non-missing part is larger than a number of pixels in said missing part.

6. The pixel interpolation device as set forth in claim 5 , wherein, in the mean preserving interpolation calculation,
a sum of pixel values of the pixels other than the missing pixel, in said missing part is determined as a missing part total,
a sum of pixel values of the pixels in said non-missing part is determined as a non-missing part total, and
a difference value obtained by subtracting said missing part total from a product of a ratio of the number of pixels in said missing part to the number of pixels in said non-missing part and said non-missing part total is calculated as the pixel value of said missing pixel.

7. The pixel interpolation device as set forth in claim 2 , wherein
said non-missing part referenced in the interpolation by at least one mean preserving interpolation calculator among said plurality of mean preserving interpolation calculators comprises a first part positioned on one side of said missing pixel and a second part positioned on the other side of said missing pixel, and
when an absolute difference value between a sum of pixel values of the pixels in said first part and a sum of pixel values of the pixels in said second part, or a difference index value determined based on said absolute difference value is larger than a predetermined threshold value, said decider selects a pre-designated interpolation calculator among said second group of interpolation calculators.

8. The pixel interpolation device as set forth in claim 7 , wherein said pre-designated interpolation calculator is an interpolation calculator performing linear interpolation calculation or bi-cubic interpolation calculation.

9. The pixel interpolation device as set forth in claim 8 , wherein said linear interpolation calculation is performed referencing non-missing pixels adjacent to the missing pixel.

10. The pixel interpolation device as set forth in claim 8 , wherein said bi-cubic interpolation calculation is performed referencing first non-missing pixels adjacent to said missing pixel, and second non-missing pixels adjacent to said first non-missing pixels and positioned on the side opposite to said missing pixel.

11. An image processing device wherein
said input image is a color image represented by three or more color component values,
said image processing device comprising:
three or more pixel interpolation devices respectively provided for the color component values, each performing interpolation of the corresponding color component value,
a luminance color-difference calculator to convert color component values outputted from said three or more pixel interpolation devices into a luminance value and first and second color-difference component values;
a color-difference calculator to convert the color component values of said input image into first and second color-difference component values;
a clip value determiner to determine at least one clip value from the first and second color-difference component values obtained by the conversion at said color-difference calculator;
a clip processor to limit the first and second color-difference component values obtained by the conversion at said luminance color-difference calculator, using said at least one clip value determined by said clip value determiner; and
a color component value recoverer to convert the first and second color-difference component values limited by said clip processor, and the luminance value obtained by the conversion at said luminance color-difference calculator, into color component values, wherein
each of said pixel interpolation devices is the pixel interpolation device as set forth in claim 1 , and performs interpolation using the corresponding color component values as the pixel values.

12. The image processing device as set forth in claim 11 , wherein said at least one clip value determined by said clip value determiner includes an upper clip value and a lower clip value.

13. The image processing device as set forth in claim 11 , wherein said clip value determiner determines said at least one clip value from the first and second color-difference comp

Assignees

• Mitsubishi Electric Corp

Inventors

• Aoki Toru
• Sugano Miki
• Asamura Masako
• Toyoda Yoshitaka
• Kurihara Kohei
• Suzuki Daisuke

Classifications

• G06T5/70 Primary

  Denoising; Smoothing · CPC title

• G06T5/77

  Retouching; Inpainting; Scratch removal · CPC title

• G06T5/002 Primary

  Physics · mapped topic

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• H04N23/843

  Demosaicing, e.g. interpolating colour pixel values · CPC title