Automatic composition of composite images or videos from frames captured with moving camera

Patent metadata

Field	Value
Publication number	US-11095833-B2
Application number	US-202016833185-A
Country	US
Kind code	B2
Filing date	Mar 27, 2020
Priority date	Sep 28, 2015
Publication date	Aug 17, 2021
Grant date	Aug 17, 2021

Abstract

A processing device generates composite images from a sequence of images. The composite images may be used as frames of video. A foreground/background segmentation is performed at selected frames to extract a plurality of foreground object images depicting a foreground object at different locations as it moves across a scene. The foreground object images are stored to a foreground object list. The foreground object images in the foreground object list are overlaid onto subsequent video frames that follow the respective frames from which they were extracted, thereby generating a composite video.

First claim

The invention claimed is:

1. A non-transitory computer-readable apparatus comprising a storage medium, the storage medium comprising a plurality of executable instructions configured to, when executed by a processor apparatus associated with a computerized apparatus, cause the computerized apparatus to:
receive a plurality of frames of a sequence obtained via a camera experiencing motion, the plurality of frames including an object, the plurality of frames including a current frame onto which one or more foreground objects are overlaid to generate a composite image;
cause performance of foreground-background processing on individual ones of one or more frames of the plurality of frames to obtain one or more foreground objects;
obtain motion information of the camera for the one or more frames, the motion information of the camera generated by a motion sensor integrated with the camera;
determine one or more relative positions of the camera for the one or more frames with respect to a current position of the camera for the current frame based on the motion information of the camera generated by the motion sensor integrated with the camera;
generate one or more transformed foreground objects by transforming the one or more foreground objects based on the one or more relative positions of the camera for the one or more frames with respect to the current position of the camera for the current frame as determined from the motion information of the camera generated by the motion sensor integrated with the camera, wherein the transforming of the one or more foreground object images includes applying a partial transparency to at least one of the one or more foreground object images; and
overlay the one or more transformed foreground objects onto the current frame to produce the composite image.

2. The non-transitory computer-readable apparatus of claim 1 , wherein the plurality of instructions are further configured to, when executed, cause application of a lens distortion effect to at least a portion of at least one foreground object, the lens distortion effect configured to compensate for a difference created by different regions of an image corresponding to different portions of a lens used to capture the plurality of frames.

3. The non-transitory computer-readable apparatus of claim 1 , wherein the plurality of executable instructions are further configured to, when executed, cause the computerized apparatus to:
obtain a plurality of candidate foreground objects from the foreground-background processing, the foreground-background processing being performed on a first range of frames from the plurality of frames, wherein the one or more foreground objects is obtained via selection from the plurality of candidate foreground objects.

4. The non-transitory computer-readable apparatus of claim 3 , wherein the plurality of executable instructions are further configured to, when executed, cause the computerized apparatus to:
store the one or more foreground objects to a foreground object list, the foreground object list being configured to store a plurality of foreground objects selected from at least a second range of frames from the plurality of frames; and
produce a composite video based at least on a plurality of foreground objects from the foreground object list, the composite video comprising the composite image.

5. A method for generating a composite output image of an object from a plurality of image frames of a sequence obtained via a camera experiencing motion, the plurality of image frames including a current frame onto which foreground images of the object are overlaid to generate the composite output image, the method comprising:
generating one or more foreground object images of the object from individual ones of one or more image frames of the plurality of image frames based on a foreground-background segmentation on the one or more image frames;
obtaining motion information of the camera for the one or more image frames, the motion information of the camera generated by a motion sensor integrated with the camera;
determine one or more relative positions of the camera for the one or more image frames with respect to a current position of the camera for the current frame based on the motion information of the camera generated by the motion sensor integrated with the camera;
generating one or more transformed foreground object images by transforming the one or more foreground object images based on the one or more relative positions of the camera for the one or more image frames with respect to the current position of the camera for the current frame as determined from the motion information of the camera generated by the motion sensor integrated with the camera, wherein the transforming of the one or more foreground object images includes applying a partial transparency to at least one of the one or more foreground object images;
generating the composite output image based at least on an overlay of the one or more transformed foreground object images onto the current frame.

6. The method of claim 5 , wherein the transforming of the one or more foreground object images further comprises a rotation operation associated with a motion with respect to a reference.

7. The method of claim 6 , wherein:
the transforming of the one or more foreground object images is performed based on an inverse of motion of the camera between the one or more relative positions of the camera for the one or more image frames with respect to the current position of the camera for the current frame.

8. The method of claim 5 , wherein the transforming of one or more foreground object images further comprises translating the one or more foreground object images based on a change in a pointing orientation of the camera.

9. The method of claim 5 , further comprising generating a set of candidate foreground object images based at least on the performing of the foreground-background segmentation, the generating of the one or more foreground object images comprising selecting the one or more foreground object images from the set;
wherein the selecting of individual ones of the one or more foreground object images occurs at least once every prescribed range of frames.

10. The method of claim 5 , wherein the generating of the one or more foreground object images is based on one or more selection criteria, the one or more selection criteria relating to one or more of a quality metric, a likelihood of feature detection, or a parameter related to motion of the camera.

11. A camera comprising:
one or more camera lenses configured to capture image content, the image content comprising data representative of a plurality of frames of a sequence captured via the camera experiencing motion, the plurality of frames including an object, the plurality of frames including a current frame onto which one or more foreground objects are overlaid to generate a composite image;
processor apparatus; and
a non-transitory computer-readable apparatus comprising a storage medium, the storage medium in data communication with the processor apparatus and comprising a plurality of executable instructions configured to, when executed by the processor apparatus, cause the computerized apparatus to:
perform foreground-background processing on individual ones of one or more frame of plurality of frames;
extract one or more foreground objects based at least on the foreground-background processing;
obtain motion information of the camera for the one or more frames, the motion information of the camera generated by a motion sensor integrated with the camera;
determine one or more relative positions of the camera for the one or more frames with resp

Assignees

• Gopro Inc

Inventors

• Adsumilli Balineedu Chowdary
• Lustig Ryan

Classifications

• H04N23/743

  Bracketing, i.e. taking a series of images with varying exposure conditions · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• H04N5/272 Primary

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title

• G06T2207/20036

  Morphological image processing · CPC title

• H04N2013/0085

  Motion estimation from stereoscopic image signals · CPC title