Image generation apparatus and image generation method

Patent metadata

Field	Value
Publication number	US-9729788-B2
Application number	US-201114354692-A
Country	US
Kind code	B2
Filing date	Nov 7, 2011
Priority date	Nov 7, 2011
Publication date	Aug 8, 2017
Grant date	Aug 8, 2017

Abstract

A characteristic point extraction unit extracts a characteristic point in an overlapping region of a plurality of component images picked up using a camera built in portable equipment and having pan angles or elevation angles different from each other. A self-position estimation unit estimates three-dimensional coordinate values of the characteristic point and three-dimensional coordinate values of a camera position, which are associated with each other, at the same time in two component images in which the camera positions are displaced from each other. A panoramic image generation unit adjusts, based on the estimated three-dimensional coordinate values of the camera position, the three-dimensional coordinate values of the characteristic point of the component image to correct the component image and combine a plurality of component images having pan angles or elevation angles different from each other to generate a synthesis image.

First claim

The invention claimed is:

1. An image generation apparatus, comprising:
a characteristic point extraction circuit configured to extract a plurality of characteristic points in an overlapping region of a plurality of component images, the plurality component images obtained by capturing a plurality of respective sequential images using a camera built in portable equipment, where each of the plurality of component images have a different pan angle or elevation angle relating to the respective sequential image, and a respective posture information being a three-axis inclination of the portable equipment relating to the respective sequential image;
a self-position estimation circuit configured to estimate, for each of two sequential component images in which camera positions are displaced from one another, a plurality of respective three-dimensional coordinate values of the respective characteristic points in the overlapping region and respective three-dimensional coordinate value of the camera position;
a component image generation circuit configured to associate, for each of the two sequential component images, the posture information, the plurality of respective three-dimensional coordinate values of the respective characteristic points and the three-dimensional coordinate values of the camera positions;
a synthesis image generation circuit configured to adjust, based on the estimated three-dimensional coordinate values of the camera positions, the plurality of respective three-dimensional coordinate values of the respective characteristic points of the component images to correct the component images and to combine the two component images to generate a panoramic image; and
an image pickup guide portion configured to display a locus of movement of the camera position based on the respective three-dimensional coordinate values of the camera position estimated by the self-position estimation circuit.

2. The image generation apparatus according to claim 1 , further comprising
a camera position displacement decision circuit configured to decide whether or not the displacement of the camera position is within a permissible range based on the respective three-dimensional coordinate values of the camera position estimated by the self-position estimation circuit;
when the displacement of the camera position exceeds the permissible range, the image pickup guide portion instructing a user to correct the camera position.

3. The image generation apparatus according to claim 2 , wherein the camera position displacement decision circuit sets a threshold value for deciding whether or not the displacement of the camera position is within the permissible range and sets the threshold value such that the threshold value has a higher value as a distance to a subject determined from the plurality of respective three-dimensional coordinate values of the respective characteristic points estimated by the self-position estimation circuit and the respective three-dimensional coordinate values of the camera position increases.

4. The image generation apparatus according to claim 3 , wherein the camera position displacement decision circuit sets, as the threshold value, a first threshold value and a second threshold value higher than the first threshold value; and
the image pickup guide portion:
(a) does not instruct, when it is decided that the displacement of the camera position is lower than the first threshold value, the user to correct the camera position but permits pickup of a component image at the camera position,
(b) instructs, when it is decided that the displacement of the camera position is equal to or higher than the first threshold value but is lower than the second threshold value, the user to correct the camera position and permits pickup of a component image at the camera position, and
(c) instructs, when the displacement of the camera position is equal to or higher than the second threshold value, the user to correct the camera position but does not permit pickup of a component image at the camera position until the camera position becomes at least lower than the second threshold value.

5. The image generation apparatus according to claim 4 , wherein the first threshold value is an upper limit to a permissible displacement amount for the camera position with which the component images can be combined to generate a synthesis image even if the component images are not corrected, and the second threshold value is an upper limit to a permissible displacement amount for the camera position with which, if the component images are corrected, then the component images after the correction can be combined to generate a synthesis image; and
the synthesis image generation circuit corrects and combines, when it is decided by the camera position displacement decision circuit that the displacement of the camera position is equal to or higher than the first threshold value but is lower than the second threshold value, the component images to generate a synthesis image.

6. An image generation method, comprising:
extracting a plurality of characteristic points in an overlapping region of a plurality of component images, the plurality component images obtained by capturing a plurality of respective sequential images using a camera built in portable equipment, where each of the plurality of component images have a different pan angle or elevation angle relating to the respective sequential image, and a respective posture information being a three-axis inclination of the portable equipment relating to the respective sequential image;
estimating, for each of two sequential component images in which camera positions are displaced from one another, a plurality of respective three-dimensional coordinate values of the respective characteristic points in the overlapping region and respective three-dimensional coordinate value of the camera position;
associating, for each of the two sequential component images, the posture information, the plurality of respective three-dimensional coordinate values of the respective characteristic points and the three-dimensional coordinate values of the camera positions;
adjusting, based on the estimated three-dimensional coordinate values of the camera positions, the plurality of respective three-dimensional coordinate values of the respective characteristic points of the component images to correct the component images and to combine the two component images to generate a panoramic image; and
displaying a locus of movement of the camera position based on the respective three-dimensional coordinate values of the camera position estimated by the self-position estimation circuit.

7. A non-transitory, computer-readable recording medium storing a program, the program causing a computer to carry out actions, comprising:
extracting a plurality of characteristic points in an overlapping region of a plurality of component images, the plurality component images obtained by capturing a plurality of respective sequential images using a camera built in portable equipment, where each of the plurality of component images have a different pan angle or elevation angle relating to the respective sequential image, and a respective posture information being a three-axis inclination of the portable equipment relating to the respective sequential image;
estimating, for each of two sequential component images in which camera positions are displaced from one another, a plurality of respective three-dimensional coordinate values of the respective characteristic points in the overlapping region and respective three-dimensional coordinate value of the camera position;
associating, for each of the two sequential component images, the posture information, the plurality of respective three-dimensio

Assignees

• Furumura Kyoko
• Hirata Shinichi
• Yamagishi Takeshi
• Sony Corp
• Sony Interactive Entertainment Inc

Inventors

• Furumura Kyoko
• Hirata Shinichi
• Yamagishi Takeshi

Classifications

• H04N23/63

  by using electronic viewfinders · CPC title

• G03B37/02 Primary

  with scanning movement of lens or cameras · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06T3/4038

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06T7/73

  using feature-based methods · CPC title