Image alignment using a virtual gyroscope model

Patent metadata

Field	Value
Publication number	US-11503209-B2
Application number	US-201916538641-A
Country	US
Kind code	B2
Filing date	Aug 12, 2019
Priority date	Jun 20, 2016
Publication date	Nov 15, 2022
Grant date	Nov 15, 2022

Abstract

A target image captured from a fisheye lens or other lens with known distortion parameters may be transformed to align it to a reference image. Corresponding features may be detected in the target image and the reference image. The features may be transformed to a spherical coordinate space. In the spherical space, images may be re-pointed or rotated in three dimensions to align all or a subset of the features of the target image to the corresponding features of the reference image. For example, in a sequence of images, background features of the target image in the spherical image space may be aligned to background features of the reference image in the spherical image space to compensate for camera motion while preserving foreground motion. An inverse transformation may then be applied to bring the images back into the original image space.

First claim

The invention claimed is:

1. A method of stabilizing content, the method comprising:
obtaining data relating to a first image and data relating to a second image, the first and second images previously captured via an imaging apparatus;
detecting a misalignment between one or more features associated with the first image and corresponding one or more features associated with the second image, the first and second images each having a first type of coordinates;
processing the data relating to the first image to produce a first transformed image, and processing the data relating to the second image to produce a second transformed image, the first and second transformed images each having a second type of coordinates;
processing the first transformed image so as to cause alignment of one or more features of the first transformed image with corresponding ones of one or more features of the second transformed image, wherein filtering is applied to retain intentional movement of the imaging apparatus and removing unintentional movement of the imaging apparatus; and
further processing the processed first transformed image to generate a processed first transformed image in the first type of coordinates.

2. The method of claim 1 , further comprising receiving video content data, the video content data comprising data relating to a sequence of images, the data relating to the sequence of images comprising the data relating to the first image and the data relating to the second image.

3. The method of claim 2 , wherein the data relating to the first image is temporally located subsequent to the data relating to the second image within the video content data.

4. The method of claim 1 , wherein the first type of coordinates comprises two-dimensional coordinates, and the second type of coordinates comprises three-dimensional coordinates.

5. The method of claim 1 , further comprising detecting an amount of at least rotational motion of the imaging apparatus, the detecting of the amount of at least rotational motion being based at least on clustering of at least a set of feature tracks between (i) the one or more features of the first transformed image and (ii) the corresponding one or more features of the second transformed image.

6. The method of claim 1 , further comprising generating stabilized video content data, the generating of the stabilized video content comprising applying a smoothing filter to at least the first and second transformed images.

7. The method of claim 1 , wherein the processing the first transformed image so as to cause alignment of one or more features of the first transformed image with corresponding ones of one or more features of the second transformed image comprises applying one or more operations to the first transformed image, the one or more operations comprising one or more of a rotation or a translation.

8. The method of claim 1 , wherein the obtaining of the data relating to the first image and the data relating to the second image comprises receiving, via a data network, the data relating to the first image and the data relating to the second image from at least one camera device configured to perform data communication with the data network.

9. A computerized apparatus configured to stabilize captured image content, the computerized apparatus comprising:
processor apparatus;
a data interface in data communication with the processor apparatus and configured to enable data communication with at least one computerized device; and
non-transitory computer-readable apparatus in data communication with the processor apparatus and comprising a storage medium, the storage medium comprising a plurality of instructions configured to, when executed by the processor apparatus, cause the imaging apparatus to:
retrieve stored content data, the content data comprising a sequence of images previously obtained from the at least one computerized device via the data interface;
determine a motion of the at least one computerized device based at least on a difference between a first image of the sequence of images and a second image of the sequence of images;
perform a first transformation on the first and second images to produce first and second transformed images;
perform a second transformation on the first transformed image with respect to the second transformed image including application of a filter to preserve an intentional motion of the determined motion, the second transformation producing a first aligned image; and
perform a third transformation on the first aligned image,
wherein the first transformation and the third transformation are performed in a coordinate system and the second transformation is performed in another coordinate system.

10. The computerized apparatus of claim 9 , wherein the plurality of instructions are further configured to, when executed by the processor apparatus, cause the computerized apparatus to adjust the stored content data based at least on the first, second, and third transformations; and
wherein the adjustment of the stored content data comprises a compensation for at least a portion of the determined motion.

11. The computerized apparatus of claim 9 , wherein the second transformation comprises at least one of a rotation or a translation of the first transformed image so as to enhance alignment of one or more features of the first transformed image with one or more corresponding features of the second transformed image.

12. The computerized apparatus of claim 9 , wherein the plurality of instructions are further configured to, when executed by the processor apparatus, cause the computerized apparatus to:
generate a difference image based at least on the second transformed image and the first aligned image; and
based at least on a threshold amount of difference corresponding to at least portions of the difference image, identify at least a foreground portion and at least a background portion.

13. The computerized apparatus of claim 12 , wherein the identification of the at least foreground portion comprises identification of a first set of feature tracks, at least some of the first set of feature tracks having different lengths as compared to at least some of a second set of feature tracks, the determined motion comprising at least a movement of one or more features of the at least foreground portion.

14. The imaging apparatus of claim 9 , wherein the at least one computerized device comprises one or more lenses, the one or more lenses comprising at least one fisheye lens having a lens-distortion parameter associated therewith.

15. The imaging apparatus of claim 14 , wherein:
the first transformation comprises an algorithmic conversion of two-dimensional coordinates of the first and second images to three-dimensional coordinates based at least on the lens-distortion parameter; and
the three-dimensional coordinates comprise a radial distance, an azimuthal angle, and an elevation angle.

16. A non-transitory computer-readable apparatus comprising a storage medium, the storage medium having a computer program comprising instructions configured to, when executed by a processor apparatus of a computerized apparatus, cause the computerized apparatus to:
obtain media content data including a first frame and a second frame, each of the first and second frames comprising a first coordinate system, the media content data generated by a remotely disposed apparatus;
transfer and align at least the first and second frames according to a second coordinate system; and
apply a correction to the media content based at least on data relating to the a

Assignees

• Gopro Inc

Inventors

• Lustig Ryan
• Adsumilli Balineedu Chowdary

Classifications

• H04N23/685

  performed by mechanical compensation · CPC title

• H04N23/6815

  by distinguishing pan or tilt from motion · CPC title

• H04N23/6812 Primary

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N5/23261

  Electricity · mapped topic