Omnidirectional moving image processing apparatus, system, method, and recording medium

Patent metadata

Field	Value
Publication number	US-11019262-B2
Application number	US-201816753910-A
Country	US
Kind code	B2
Filing date	Nov 12, 2018
Priority date	Dec 18, 2017
Publication date	May 25, 2021
Grant date	May 25, 2021

Abstract

An image processing apparatus including a storage unit, a correction unit, and an output unit. The storage unit stores moving-image data including a plurality of frames captured by an image-capturing device communicable with the image processing apparatus, time-series data of an inclination angle with reference to a reference direction of the image-capturing device, and time-series data of an angle velocity of the image-capturing device. The correction unit, based on the time-series data of the angle velocity, rotates an image of each of the plurality of frames of the moving-image data to reduce a rotational distortion around the reference direction within a prescribed frequency range. The output unit outputs image data of the rotated image of each of the plurality of frames to an external device communicable with the image processing apparatus.

First claim

The invention claimed is:

1. An image processing apparatus, comprising:
a memory to store moving-image data including a plurality of frames captured by an image-capturing device communicable with the image processing apparatus, time-series data of an inclination angle of the image-capturing device with reference to a reference direction, wherein the reference direction is a direction of gravity, and time-series data of an angular velocity of the image-capturing device; and
circuitry configured to perform a zenith correction on an omnidirectional image of each of the plurality of frames of the moving-image data based on the time-series data of the inclination angle, and, based on the time-series data of the angular velocity, perform rotation correction to rotate the zenith-corrected omnidirectional image of each of the plurality of frames of the moving-image data by a calculated correction angle within a plane perpendicular to the reference direction to reduce a rotational distortion around the reference direction within a prescribed frequency range, the correction angle being calculated based on both the times-series data of the inclination angle and the time-series data of the angular velocity, and output image data of the rotated omnidirectional image of each of the plurality of frames to an external device communicable with the image processing apparatus.

2. The image processing apparatus according to claim 1 ,
wherein, in response to a request, from the external device, to output a moving image of the rotated image of each of the plurality of frames, the circuitry is further configured to start rotating an image of each of the plurality of frames of the moving-image data to output image data of the rotated image of each of the plurality of frames.

3. The image processing apparatus according to claim 1 , wherein the circuitry is further configured to:
calculate an initial front direction of the image-capturing device at an initial time based on a value of the time-series data of the inclination angle at the initial time;
calculate time-series values of a front direction of the image-capturing device over a plurality of times after the initial time and corresponding to the plurality of frames, based on the calculated initial front direction and the time-series data of the angular velocity; and
calculate the correction angle at each of the plurality of times in rotating the image from the calculated front direction at each of the plurality of times, based on the time-series data of the inclination angle,
wherein each front direction is a direction of the optical axis of the image-capturing device at a corresponding time of the plurality of times.

4. The image processing apparatus according to claim 3 , further comprising a filter configured to perform a filtering process on the calculated correction angle at each of the plurality of times to allow for passage of a high-frequency component within the prescribed frequency range.

5. The image processing apparatus according to claim 3 , wherein the circuitry is further configured to detect a change of the correction angle from a first limit of a predetermined domain of the correction angle to a second limit of the predetermined domain of the correction angle, and to add or subtract an adjustment value to or from the correction angle.

6. The image processing apparatus according to claim 3 , wherein the circuitry is further configured to obtain the front direction at each of the plurality of times by integrating an infinitely small rotation corresponding to the angular velocity at each axis of the time-series data of the angular velocity, starting from the initial front direction at the initial time.

7. An image processing system, comprising:
a memory to store moving-image data including a plurality of frames captured by an image-capturing device communicable with an image processing apparatus, time-series data of an inclination angle with reference to a reference direction of the image-capturing device, wherein the reference direction is a direction of gravity, and time-series data of an angular velocity of the image-capturing device; and
circuitry configured to perform a zenith correction on an omnidirectional image of each of the plurality of frames of the moving-image data based on the time-series data of the inclination angle, and, based on the time-series data of the angular velocity, perform rotation correction to rotate the zenith-corrected omnidirectional image of each of the plurality of frames of the moving-image data by a calculated correction angle within a plane perpendicular to the reference direction to reduce a rotational distortion around the reference direction within a prescribed frequency range, the correction angle being calculated based on both the times-series data of the inclination angle and the time-series data of the angular velocity, and output image data of the rotated omnidirectional image of each of the plurality of frames to an external device communicable with the image processing apparatus.

8. The image processing system according to claim 7 , further comprising:
the image-capturing device including the memory, and the circuitry; and
an information processing apparatus communicable with the image-capturing device, the information processing apparatus comprising:
a receiver to receive a request to output the rotated moving-image;
a display to display image data output from the image-capturing device;
wherein, in response to a request, from the external device, to output a moving image of the rotated image of each of the plurality of frames, the circuitry is further configured to start rotating an image of each of the plurality of frames of the moving-image data to output image data of the rotated image of each of the plurality of frames.

9. A method of processing an image, the method comprising:
storing moving-image data including a plurality of frames captured by an image-capturing device communicable with an image processing apparatus, time-series data of an inclination angle with reference to a reference direction of the image-capturing device, wherein the reference direction is a direction of gravity, and time-series data of an angular velocity of the image-capturing device;
performing a zenith correction on an omnidirectional image of each of the plurality of frames of the moving-image data based on the time-series data of the inclination angle;
based on the time-series data of the angular velocity, performing rotation correction to rotate the zenith-corrected omnidirectional image of each of the plurality of frames of the moving-image data by a calculated correction angle within a plane perpendicular to the reference direction to reduce a rotational distortion around the reference direction within a prescribed frequency range, the correction angle being calculated based on both the times-series data of the inclination angle and the time-series data of the angular velocity;
outputting image data of the rotated omnidirectional image of each of the plurality of frames to an external device communicable with the image processing apparatus.

10. The image processing apparatus of claim 1 , wherein
the zenith correction performed by the circuitry corrects each omnidirectional image of the plurality of frames in roll and pitch directions with respect to the direction of gravity, based on the time-series data of the inclination angle; and
the rotation correction performed by the circuitry corrects each omnidirectional image of the plurality of frames in a yaw direction with respect to the direction of gravity, based on the calculated correction angle, to reduce the rotational distortion.

Assignees

• Kawasaki Gaku
• Ricoh Co Ltd

Inventors

• Kawasaki Gaku

Classifications

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/81

  for suppressing or minimising disturbance in the image signal generation · CPC title

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• G06T3/60 Primary

  Rotation of whole images or parts thereof · CPC title

• H04N5/2628

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title