Image signal processor and method for synthesizing super-resolution images from non-linear distorted images

Patent metadata

Field	Value
Publication number	US-9609181-B2
Application number	US-201113992807-A
Country	US
Kind code	B2
Filing date	Dec 20, 2011
Priority date	Dec 27, 2010
Publication date	Mar 28, 2017
Grant date	Mar 28, 2017

Abstract

An image signal processing apparatus and image signal processing method obtains a high super-resolution effect for distorted images captured by a specialized optical system, such as a 360-degree camera. A distortion-correction conversion-formula calculation section calculates a distortion correction conversion formula for correcting the nonlinear distortion. A distortion correction and interpolation processing section performs distortion correction and interpolation processing on each cropped region. An alignment conversion formula calculation section calculates an alignment conversion formula for alignment between each cropped image after distortion correction and interpolation processing is completed. A pixel coordinate conversion section rearranges pixel coordinates for each frame image before interpolation processing by using the distortion correction conversion formula and the alignment conversion formula corresponding to the cropped regions. An image synthesis section synthesizes the cropped images.

First claim

The invention claimed is:

1. A video signal processing apparatus, comprising:
a first calculator that receives a plurality of original input image frames, each of which having a different non-linear distortion, and calculates first coordinate conversion formulas, corresponding to each of the original input image frames, correcting non-linear distortion corresponding to a cropped region, capturing an object of each of the frame images;
a distortion correction and interpolation processor that generates a plurality of distortion corrected images by performing distortion correction on the cropped region of each of the original input frame images according to each of the first coordinate conversion formulas and generates a plurality of interpolation processed images by performing interpolation processing on each of the distortion corrected images;
a second calculator that calculates second coordinate conversion formulas correcting the interpolation processed images so that a position of the object captured in each of the interpolation processed images matches each other;
an image coordinate conversion formula calculator that calculates integrated coordinate conversion formulas by integrating the first coordinate conversion formulas and the second coordinate conversion formulas;
a pixel coordinate converter that generates pixel rearranged images by performing rearrangement of pixel coordinates on at least the cropped region of each of the original input frame images using the integrated coordinate conversion formula corresponding to each of the original input image frames; and
a synthesizer that generates a synthesized image by synthesizing the rearranged images.

2. The video signal processing apparatus according to claim 1 , further comprising:
an interpolation processor that performs interpolation processing on the synthesized image obtained by the synthesizer.

3. A camera system, comprising:
a camera that photographs an image having non-linear distortion; and
the video signal processing apparatus according to claim 1 that performs processing on the photographed image.

4. A video signal processing method, comprising:
receiving a plurality of original input image frames, each having a different non-linear distortion;
calculating first coordinate formulas, corresponding to each of the original input image frames, correcting non-linear distortion corresponding to a cropped region, capturing an object, of each of the frame images;
generating a plurality of distortion corrected images by performing distortion correction on the cropped region of each of the original input frame images according to each of the first coordinate conversion formulas, and generating a plurality of interpolation processed images by performing interpolation processing on each of the distortion corrected images;
calculating second coordinate conversion formulas correcting the interpolation processed images so that pixel positions in the object captured correspond with pixel positions in each of the interpolation processed images;
calculating integrated coordinate conversion formulas by integrating the first coordinate conversion formulas and the second coordinate conversion formulas;
generating pixel rearranged images by performing rearrangement of pixel coordinates on at least the cropped region of each of the original input frame images using the integrated coordinate conversion formula corresponding to each of the original input image frames; and
generating a synthesized image by synthesizing the rearranged images.

5. The video signal processing apparatus according to claim 1 , further comprising:
a region cropper that decides the cropped region corresponding to at least a portion of each of the original input image frames so that the cropped region captures the object.

6. The video signal processing apparatus according to claim 5 ,
wherein the region cropper performs tracking of the object captured in each of the original input image frames captured by a camera at different times.

7. The video signal processing apparatus according to claim 6 ,
wherein the region cropper decides a position of the cropped region based on the tracking.

8. The video signal processing apparatus according to claim 5 ,
wherein the region cropper decides a position of the cropped region in response to a user input.

9. The video signal processing apparatus according to claim 1 , further comprising:
a memory that stores parameters for calculating a distortion corresponding to each of positions of the original input image frames.

10. The video signal processing apparatus according to claim 1 ,
wherein the first calculator calculates the first coordinate conversion formula based on position information of the cropped region and parameters for calculating a distortion corresponding to each of positions of the original input image frame.

11. The video signal processing apparatus according to claim 1 , further comprising:
an output that outputs the synthesized image or an image further processed based on the synthesized image.

12. The video signal processing apparatus according to claim 1 ,
wherein the pixel coordinate convertor performs (1) distortion correction according to the first coordinate conversion formulas, and (2) rearrangement of pixel coordinates according to the second coordinate conversion formulas, on the cropped region corresponding to each of the original input frame images.

13. A video signal processing apparatus, comprising:
a first calculator that receives a plurality of original input image frames, each of which having a different non-linear distortion, and calculates first coordinate conversion formulas, corresponding to each of the original input image frames, correcting non-linear distortion corresponding to a cropped region, capturing an object of each of the frame images;
a distortion correction and interpolation processor that generates a plurality of distortion corrected images by performing distortion correction on the cropped region of each of the original input frame images according to each of the first coordinate conversion formulas and generates a plurality of interpolation processed images by performing interpolation processing on each of the distortion corrected images;
a second calculator that calculates second coordinate conversion formulas correcting the interpolation processed images so that a position of the object captured in each of the interpolation processed images matches each other;
a pixel coordinate converter that generates pixel rearranged images by performing rearrangement of pixel coordinates on at least the cropped region of each of the original input frame images using the first coordinate conversion formula for correcting each non-linear distortion of the original input image frames and the second coordinate conversion formula for correcting the original input image frames so that a position of the object captured in each of the original input image frames matches each other; and
a synthesizer that generates a synthesized image by synthesizing the rearranged images.

Assignees

• Wang Qinhe
• Tabei Kenji
• Panasonic Ip Man Co Ltd

Inventors

• Wang Qinhe
• Tabei Kenji

Classifications

• H04N1/407 Primary

  Control or modification of tonal gradation or of extreme levels, e.g. background level · CPC title

• H04N5/2628

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06T3/4053

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• H04N1/387 Primary

  Composing, repositioning or otherwise {geometrically} modifying originals · CPC title