Automatic composition of composite images or videos from frames captured with moving camera

Patent metadata

Field	Value
Publication number	US-10609307-B2
Application number	US-201615080299-A
Country	US
Kind code	B2
Filing date	Mar 24, 2016
Priority date	Sep 28, 2015
Publication date	Mar 31, 2020
Grant date	Mar 31, 2020

Abstract

A processing device generates composite images from a sequence of images. The composite images may be used as frames of video. A foreground/background segmentation is performed at selected frames to extract a plurality of foreground object images depicting a foreground object at different locations as it moves across a scene. The foreground object images are stored to a foreground object list. The foreground object images in the foreground object list are overlaid onto subsequent video frames that follow the respective frames from which they were extracted, thereby generating a composite video.

First claim

The invention claimed is:

1. A method for generating a composite output image from an input video having a sequence of frames, the method comprising:
initializing a predictive model based at least on a plurality of training images;
receiving a sequence of image frames depicting a foreground object, the sequence of image frames captured by a camera experiencing motion;
for selected frames comprising a first quantity of frames in the sequence of image frames, (i) performing by a processing device, foreground/background segmentations to extract, based at least on the predictive model, a set of respective candidate foreground object images each comprising a representation of the foreground object with background pixels subtracted, the performing of the foreground/background segmentations comprising, based at least on an image criterion associated with each of the respective candidate foreground object images, selecting a foreground object image from the set of respective candidate foreground object images, and (ii) storing the selected foreground object image to a foreground object list, the foreground object list comprising a plurality of foreground object images selected from respective sets of candidate foreground object images, the selecting of the foreground object image occurring at least once for every second quantity of frames, the second quantity of frames being greater than the first quantity of frames;
determining, for each of the selected foreground object images in the foreground object list, a respective motion between (i) a position of the camera at which a selected frame corresponding to a respective selected foreground object image was captured and (ii) a position of the camera capturing a current frame of the respective selected foreground object image;
transforming the representation of foreground objects in the selected foreground object images based on the respective motion to generate transformed foreground object images; and
overlaying the transformed foreground object images onto the current frame to generate a composite output image.

2. The method of claim 1 , wherein the transforming of the representation comprises:
translating a location of the representation of the foreground object in the respective selected foreground object images based on a component of the motion, the component of the motion representing a change in planar position of the camera between capturing the selected frame and capturing the current frame.

3. The method of claim 2 , wherein the transforming of the representation further comprises translating the location according to a motion equal and opposite to the component of the motion representing the change in planar position of the camera between capturing the selected frame and capturing the current frame.

4. The method of claim 1 , wherein the transforming of the representation comprises:
translating a location of the representation of the foreground object in the selected foreground object image based on a component of the motion, the component of the motion representing a change in rotational position of the camera about a reference point between capturing the selected frame and capturing the current frame.

5. The method of claim 4 , wherein the transforming of the representation further comprises translating the location according to a motion equal and opposite to the component of the motion representing the change in rotational position of the camera between capturing the selected frame and capturing the current frame.

6. The method of claim 1 , wherein the transforming of the representation comprises:
scaling the representation of the foreground object in the selected foreground object image based on a component of the motion, the component of the motion representing a change in depth of the camera relative to a reference plane between capturing the selected frame and capturing the current frame.

7. The method of claim 6 , wherein the transforming of the representation further comprises:
enlarging the representation of the foreground object in the selected foreground object image in response to the camera being closer to a position of the foreground object when the selected frame was captured; and
reducing in size the representation of the foreground object in the selected foreground object image in response to the camera being further from the position of the foreground object when the selected frame was captured.

8. The method of claim 1 , wherein the transforming of the representation comprises:
translating the representation of the foreground object in the selected foreground object image based on a component of the motion, the component of the motion representing a change in pointing orientation of the camera between the current frame and the selected frame.

9. The method of claim 8 , wherein the transforming of the representation further comprises:
applying a lens distortion effect to the representation of the foreground object based on the component of the motion representing the change in pointing orientation of the camera between the current frame and the selected frame.

10. The method of claim 1 , further comprising:
detecting an overlaid foreground object image exceeding an overlap threshold of at least one other overlaid foreground object image; and
applying a partial transparency to the detected overlaid foreground object image exceeding the overlap threshold.

11. The method of claim 1 , wherein the determining of the respective motion comprises:
identifying features in the selected frame corresponding to the respective selected foreground object image, and identifying corresponding features in the current frame;
performing a correlation to determine estimated motions between the identified features in the selected frame and the corresponding features in the current frame; and
determining the respective motion of the camera based on the estimated motions between the identified features in the selected frame and the corresponding features in the current frame.

12. The method of claim 1 , wherein the determining of the respective motion comprises:
obtaining motion sensor data from one or more motion sensors of the camera; and
determining the respective motion of the camera based on the motion sensor data.

13. A non-transitory computer-readable storage medium storing instructions for generating a composite output video from an input video having a sequence of frames, the instructions when executed by a processor apparatus causing the processor to perform steps comprising:
training a predictive model based at least on a plurality of input images;
receiving a sequence of image frames depicting a foreground object, the sequence of image frames captured by a camera experiencing motion;
for selected frames comprising a first quantity of frames in the sequence of image frames, (i) performing foreground/background segmentations to extract, based at least on the predictive model, a set of respective candidate foreground object images each comprising a representation of the foreground object with background pixels subtracted, wherein the performing of the foreground/background segmentations comprises selecting a foreground object image from the set of respective candidate foreground object images based on an image criterion associated with each of the respective candidate foreground object images, and (ii) storing the selected foreground object image to a foreground object list that comprises a plurality of foreground object images selected from respective sets of candidate foreground object images, where the predictive model is configured to be updated at least once for every seco

Assignees

• Gopro Inc

Inventors

• Adsumilli Balineedu Chowdary
• Lustig Ryan

Classifications

• G06T15/503

  Blending, e.g. for anti-aliasing · CPC title

• G06T7/174

  involving the use of two or more images · CPC title

• G06T7/254

  involving subtraction of images · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• H04N2013/0085

  Motion estimation from stereoscopic image signals · CPC title