Image processing device and method, and program for correcting a blurred moving image

Patent metadata

Field	Value
Publication number	US-9569823-B2
Application number	US-201314652546-A
Country	US
Kind code	B2
Filing date	Dec 12, 2013
Priority date	Dec 26, 2012
Publication date	Feb 14, 2017
Grant date	Feb 14, 2017

Abstract

Provided is an image processing device including an acquisition unit configured to acquire information on an imaging position and an imaging direction in units of frame images that constitute a moving image obtained through capturing by an imaging unit, a converted image generation unit configured to generate converted images having different imaging directions for each frame image that constitutes the moving image based on the frame image itself and preceding and succeeding frame images of the frame image, an evaluation value calculation unit configured to calculate an evaluation value for each converted moving image constituted by combining the converted image and the original frame image, the evaluation value being used to evaluate a blur between the converted images or between the original frame images, and a selection unit configured to select a converted moving image with less blur based on an evaluation value calculated by the evaluation value calculation unit.

First claim

The invention claimed is:

1. An image processing device comprising:
an acquisition unit configured to acquire information on an imaging position and an imaging direction in units of frame images that constitute a moving image obtained through capturing by an imaging unit;
a converted image generation unit configured to generate a plurality of converted images having different imaging directions for each frame image that constitutes the moving image based on the frame image itself and preceding and succeeding frame images of the frame image;
an evaluation value calculation unit configured to calculate an evaluation value for each converted moving image constituted by combining the converted image and the original frame image, the evaluation value being used to evaluate a blur between the converted images or between the original frame images; and
a selection unit configured to select a converted moving image with less blur based on an evaluation value calculated by the evaluation value calculation unit,
wherein the acquisition unit, the converted image generation unit, the evaluation value calculation unit, and the selection unit are each implemented via at least one processor.

2. The image processing device according to claim 1 , further comprising:
a node difference calculation unit configured to calculate a difference between the converted image generated by the converted image generation unit and the original frame image as a node difference; and
an edge difference calculation unit configured to calculate a difference in a time series including the converted image generated by the converted image generation unit and the original frame image as an edge difference,
wherein the evaluation value calculation unit calculates an evaluation value used to evaluate a blur for each of converted moving images based on each of a sum of the edge difference and a sum of the node difference between the converted images or the original frame images included in the converted moving image constituted by combining the converted image and the original image, and
wherein the node difference calculation unit and the edge difference calculation unit are each implemented via at least one processor.

3. The image processing device according to claim 2 ,
wherein the evaluation value calculation unit is further configured to calculate an evaluation value used to evaluate a blur for each of the converted moving images by applying a weight to each of the sum of the edge difference and the sum of the node difference between the converted images or the original frame images included in the converted moving image constituted by combination of the converted image and the original image and by calculating a sum of the weighted values.

4. The image processing device according to claim 3 , further comprising:
a weight input unit configured to input a weight to be applied to the sum of the edge difference and the sum of the node difference, a sum of the weight applied to each of the sum of the edge difference and the sum of the node difference being a value of 1,
wherein the evaluation value calculation unit calculates an evaluation value used to evaluate a blur for each of the converted moving images by applying a weight inputted by the weight input unit to each of the sum of the edge difference and the sum of the node difference between the converted images or the original frame images included in the converted moving image constituted by combination of the converted image and the original image and by calculating a sum of the weighted values, and
wherein the weight input unit is implemented via at least one processor.

5. The image processing device according to claim 1 ,
wherein the storage unit is further configured to store information on an imaging direction being changed in accordance with the original frame image by having a plurality of predetermined angles of view and by rotating to a plurality of angles in association with the converted image when the converted image is generated by the converted image generation unit,
wherein the image processing device further comprises a display image generation unit configured to generate a display image that displays a route on a map based on the converted moving image or the imaging position information of the frame image of the moving image and displays by applying a different color to each imaging direction on the route based on the imaging direction information, and
wherein the display image generation unit is implemented via at least one processor.

6. The image processing device according to claim 5 , further comprising:
a specifying unit configured to specify a position on the route displayed on the map on the display image; and
a reproduction unit configured to reproduce the converted moving image that constitutes the route or the moving image from a frame image including information on an imaging position corresponding to a position specified by the specifying unit,
wherein the specifying unit and the reproduction unit are each implemented via at least one processor.

7. The image processing device according to claim 1 ,
wherein the converted image generation unit is further configured to generate a converted image by converting each frame image that constitutes a moving image stored in the storage unit into a plurality of images having the different imaging directions by having a plurality of predetermined angles of view and by rotating to a plurality of angles by a technique including an angle of view shift, homography transformation, and viewpoint composition, using the frame image itself and preceding and succeeding frame images of the frame image.

8. The image processing device according to claim 1 ,
wherein the converted image generation unit is further configured to generate a plurality of higher-resolution converted images having the different imaging directions for each frame image that constitutes a moving image stored in the storage unit by using a pixel generated by a phase shift equal to or smaller than a pixel based on the frame image itself and preceding and succeeding frame images of the frame image.

9. The image processing device according to claim 1 ,
wherein the imaging unit is further configured to capture the moving image, and
wherein the acquisition unit is further configured to store the information on the imaging position and the imaging direction of the frame image in association with each of the frame images.

10. An image processing method, performed via at least one processor, the method comprising:
acquiring information on an imaging position and an imaging direction in units of frame images that constitute a moving image obtained through capturing by an imaging unit;
generating a plurality of converted images having different imaging directions for each frame image that constitutes the moving image based on the frame image itself and preceding and succeeding frame images of the frame image;
calculating an evaluation value for each converted moving image constituted by combining the converted image and the original frame image, the evaluation value being used to evaluate a blur between the converted images or between the original frame images; and
selecting a converted moving image with less blur based on a calculated evaluation value.

11. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer, causes the computer to execute a method, the method comprising:
acquiring information on an imaging position and an imaging direction in units of frame images that constitute a moving image obtained through capturing by an imaging unit;
generating a plurality of converted

Assignees

• Sony Corp

Inventors

• Sutou Yasuhiro
• Shimomura Hideki
• Okubo Atsushi
• Aoyama Kazumi
• Tanaka Akichika

Classifications

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/682

  Vibration or motion blur correction · CPC title

• H04N23/633

  for displaying additional information relating to control or operation of the camera · CPC title

• G03B17/48 Primary

  adapted for combination with other photographic or optical apparatus · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title