Image processing apparatus, image capturing system, image processing method, and recording medium

Patent metadata

Field	Value
Publication number	US-10855916-B2
Application number	US-201916353051-A
Country	US
Kind code	B2
Filing date	Mar 14, 2019
Priority date	Mar 15, 2018
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

An image processing apparatus obtains a first image in a first projection, and a second image in a second projection; transforms projection of a first corresponding area of the first image to generate a third image in the second projection; identifies a plurality of feature points in the second image and the third image; determines a second corresponding area in the third image, based on the plurality of feature points; generate a reference shape in the second projection, having a relation of projection transformation with the second image; transforms projection of the reference shape to generate a reference shape in the first projection; calculates reference shape conversion data for converting the reference shape to the second corresponding area, using data for transforming projection from the second image to the second corresponding area; and stores the reference shape conversion data in association with reference location data.

First claim

The invention claimed is:

1. An image processing apparatus comprising processing circuitry configured to:
obtain a first image in a first projection, and a second image in a second projection;
transform projection of a first corresponding area of the first image that corresponds to the second image, from the first projection to the second projection, to generate a third image in the second projection;
identify a plurality of feature points, respectively, in the second image and the third image;
determine a second corresponding area in the third image that corresponds to the second image, based on the plurality of feature points respectively identified in the second image and the third image;
generate a reference shape in the second projection, having a relation of projection transformation with the second image;
transform projection of the reference shape, from the second projection to the first projection, to generate a reference shape in the first projection;
calculate reference shape conversion data for converting the reference shape in the second projection to the second corresponding area, using data for transforming projection from the second image to the second corresponding area; and
store, in a memory, the reference shape conversion data in association with reference location data indicating the locations of a plurality of points of the reference shape in the first projection,
the reference shape conversion data including at least one of reference shape rotation data defining rotation processing to be applied to the reference shape, and reference shape scaling data for defining scaling to be applied to the reference shape.

2. The image processing apparatus of claim 1 , wherein the processing circuitry is configured to:
transform projection of a central point of the second corresponding area, from the second projection to the first projection, to obtain a coordinate of a corresponding point in the first image, to obtain reference shape movement data; and
store the reference shape movement data as a part of the reference shape conversion data,
the reference shape movement data defining a location of the second image in the first image.

3. The image processing apparatus of claim 2 , wherein the reference shape rotation data is represented by the Euler angle with respect to each axis of a three-dimensional model space.

4. The image processing apparatus of claim 2 , wherein the second image is a frame obtained from a plurality of frames of a video image.

5. The image processing apparatus of claim 4 , wherein the first image is a frame obtained from a plurality of frames of a video image.

6. The image processing apparatus of claim 5 , wherein the processing circuitry calculates the reference shape conversion data, based on one or more frames obtained through sampling the plurality of frames of the video image for the second image.

7. The image processing apparatus of claim 6 ,
wherein a number of the one or more frames obtained through sampling increases with an increase in the degree of change in contents of image between the frames of the video image, and decreases with a decrease in the degree of change in contents of image between the frames of the video image.

8. The image processing apparatus of claim 2 ,
wherein the reference shape is determined based on an angle of view that is initially set.

9. The image processing apparatus of claim 2 , wherein the first image is a spherical image, and the second image is a planar image.

10. The image processing apparatus of claim 2 , wherein the first projection is an equirectangular projection, and the second projection is a perspective projection.

11. The image processing apparatus of claim 1 , wherein the image processing apparatus includes at least one of a smart phone, tablet personal computer, notebook computer, desktop computer, and server computer.

12. An image capturing system comprising:
the image processing apparatus of claim 1 ;
a first image capturing device configured to capture surroundings of a target object to obtain the first image in the first projection and transmit the first image in the first projection to the image processing apparatus; and
a second image capturing device configured to capture the target object to obtain the second image in the second projection and transmit the second image in the second projection to the image processing apparatus.

13. The image capturing system of claim 12 ,
wherein the first image capturing device is a camera configured to capture the target object to generate a spherical image as the first image.

14. An image processing method, comprising:
obtaining a first image in a first projection, and a second image in a second projection;
transforming projection of a first corresponding area of the first image that corresponds to the second image, from the first projection to the second projection, to generate a third image in the second projection;
identifying a plurality of feature points, respectively, in the second image and the third image;
determining a second corresponding area in the third image that corresponds to the second image, based on the plurality of feature points respectively identified in the second image and the third image;
generating a reference shape in the second projection, having a relation of projection transformation with the second image;
transforming projection of the reference shape, from the second projection to the first projection, to generate a reference shape in the first projection;
calculating reference shape conversion data for converting the reference shape in the second projection to the second corresponding area, using data for transforming projection from the second image to the second corresponding area; and
storing, in a memory, the reference shape conversion data in association with reference location data indicating the locations of a plurality of points of the reference shape in the first projection,
the reference shape conversion data including at least one of reference shape rotation data defining rotation processing to be applied to the reference shape, and reference shape scaling data for defining scaling to be applied to the reference shape.

15. The image processing method of claim 14 , further comprising:
transforming projection of a central point of the second corresponding area, from the second projection to the first projection, to obtain a coordinate of a corresponding point in the first image, to obtain reference shape movement data; and
storing the reference shape movement data as a part of the reference shape conversion data,
the reference shape movement data defining a location of the second image in the first image.

16. A non-transitory recording medium which, when executed by one or more processors, cause the processors to perform an image processing method comprising:
obtaining a first image in a first projection, and a second image in a second projection;
transforming projection of a first corresponding area of the first image that corresponds to the second image, from the first projection to the second projection, to generate a third image in the second projection;
identifying a plurality of feature points, respectively, in the second image and the third image;
determining a second corresponding area in the third image that corresponds to the second image, based on the plurality of feature points respectively identified in the second image and the third image;
generating a reference shape in the second projection, having a relation of projection transformation with the second image;
trans

Assignees

• SUITOH Hiroshi
• Kawaguchi Keiichi
• Kawamura Yohsuke
• Ricoh Co Ltd

Inventors

• SUITOH Hiroshi
• Kawaguchi Keiichi
• Kawamura Yohsuke

Classifications

• G06F9/30 Primary

  Arrangements for executing machine instructions, e.g. instruction decode (for executing microinstructions G06F9/22) · CPC title

• H04N25/41

  Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors · CPC title

• H04N23/66

  Remote control of cameras or camera parts, e.g. by remote control devices · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/50

  Constructional details · CPC title