Automatic composition of video with dynamic background and composite frames selected based on foreground object criteria

Patent metadata

Field	Value
Publication number	US-10044944-B2
Application number	US-201615080286-A
Country	US
Kind code	B2
Filing date	Mar 24, 2016
Priority date	Sep 28, 2015
Publication date	Aug 7, 2018
Grant date	Aug 7, 2018

Abstract

A processing device generates composite images from a sequence of images. The composite images may be used as frames of video. A foreground/background segmentation is performed at selected frames to extract a plurality of foreground object images depicting a foreground object at different locations as it moves across a scene. The foreground object images are stored to a foreground object list. The foreground object images in the foreground object list are overlaid onto subsequent video frames that follow the respective frames from which they were extracted, thereby generating a composite video.

First claim

The invention claimed is:

1. A method for generating a composite output video from an input video having a sequence of frames, the method comprising:
storing a foreground object list comprising a plurality of previously extracted foreground object images, each of the foreground object images comprising a representation of a foreground object with background pixels subtracted;
receiving a current video frame for processing from the sequence of frames;
performing, by a processing device, a foreground/background segmentation based on a predictive model to extract a foreground object image for the current video frame;
determining, by the processing device, if the foreground object image meets first predefined criteria, the determining comprising:
identifying a plurality of foreground object images that each overlap a most recent foreground object image in the foreground object list by an overlap amount within a predefined overlap range;
determining a best foreground object image from among the plurality of foreground object images according to a predefined image criteria; and
determining that the foreground object image for the current video frame meets the first predefined criteria if the foreground object image for the current video frame is the best foreground object image from the plurality of foreground object images having the overlap amount within the predefined overlap range;

responsive to the foreground object image meeting the first predefined criteria, storing the foreground object image for the current frame to the foreground object list; and
overlaying each of the foreground object images in the foreground object list onto the current video frame to generate a composite video frame.

2. The method of claim 1 , wherein the determining if the foreground object image meets the first predefined criteria comprises:
determining an overlap amount between the foreground object image for the current video frame and a most recent foreground object image in the foreground object list; and
determining that the foreground object image meets the first predefined criteria if the overlap amount is less than a predefined threshold percentage.

3. The method of claim 2 , wherein the determining the overlap amount comprises determining a percentage of overlap between the foreground object image for the current video frame and a most recent foreground object image in the foreground object list; and
wherein the method further comprises, when the foreground object image meets the first predefined criteria, adding the foreground object image for the current video frame to the foreground object list.

4. The method of claim 1 , wherein the determining if the foreground object image meets the first predefined criteria comprises:
determining an image quality metric for the foreground object image; and
determining that the foreground object image meets the first predefined criteria if the image quality metric exceeds a predefined quality threshold.

5. The method of claim 1 , wherein the determining if the foreground object image meets the first predefined criteria comprises:
performing a face detection on the foreground object image; and
determining that the foreground object image meets the first predefined criteria if a face is detected.

6. The method of claim 1 , wherein the determining if the foreground object image meets the first predefined criteria comprises:
determining a motion parameter of the foreground object image; and
determining that the foreground object image meets the first predefined criteria if the motion parameter corresponds to a predefined motion threshold.

7. The method of claim 1 , further comprising:
training the predictive model based on a plurality of training video frames, the training of the predictive model comprising predicting whether a pixel in a given video frame belongs to a background model or the foreground object.

8. The method of claim 1 , further comprising:
determining if a frame number of the current video frame is a multiple of a predefined integer X; and
responsive to the frame number of the current video frame being the multiple of the predefined integer X, updating the predictive model.

9. The method of claim 1 , wherein performing the foreground/background segmentation comprises:
obtaining a preliminary foreground object image;
applying a filter to reduce noise in the preliminary foreground object image to generate a filtered image;
detecting a filled convex hull region in the preliminary foreground object image;
adding extra pixels from the filtered image to the preliminary foreground object image to generate a temporary image;
discarding pixels in the temporary image outside the filled convex hull region to generate a noisy convex hull image; and
closing gaps in foreground regions of the noisy convex hull image to generate the foreground object image.

10. The method of claim 1 , wherein the predictive model comprises an adaptive Gaussian Mixture Model.

11. A non-transitory computer-readable storage apparatus comprising a storage medium having a computer program stored thereon, the computer program comprising instructions for generating a composite output video from an input video, the input video having a sequence of frames, the instructions, when executed by a processing apparatus:
store a foreground object list comprising a plurality of previously extracted foreground object images, each of the foreground object images comprising a representation of a foreground object with background pixels subtracted;
receive a current video frame for processing from the sequence of frames;
perform a foreground/background segmentation based on a predictive model to extract a foreground object image for the current video frame;
determine if the foreground object image meets first predefined criteria, the determination comprising:
an identification of a plurality of foreground object images that each overlap a most recent foreground object image in the foreground object list by an overlap amount within a predefined overlap range;
determine a best foreground object image from among the plurality of foreground object images according to a predefined image criteria; and
determine that the foreground object image for the current video frame meets the first predefined criteria if the foreground object image for the current video frame is the best foreground object image from the plurality of foreground object images having the overlap amount within the predefined overlap range;

responsive to the foreground object image meeting the first predefined criteria, store the foreground object image for the current frame to the foreground object list; and
overlay each of the foreground object images in the foreground object list onto the current video frame to generate a composite video frame.

12. The non-transitory computer-readable storage apparatus of claim 11 , wherein the determination if the foreground object image meets the first predefined criteria further comprises:
a determination of an overlap amount between the foreground object image for the current video frame and a most recent foreground object image in the foreground object list; and
a determination that the foreground object image meets the first predefined criteria if the overlap amount is less than a predefined threshold percentage.

13. The non-transitory computer-readable storage apparatus of claim 11 , wherein the determination if the foreground object image meets the first predefined criteria further comprises:
a determination of an image quality metric for the foreground object image; and
a determination that the foreground o

Assignees

• Gopro Inc

Inventors

• Adsumilli Balineedu Chowdary
• Lustig Ryan

Classifications

• H04N23/743

  Bracketing, i.e. taking a series of images with varying exposure conditions · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/254

  involving subtraction of images · CPC title

• G06T2207/20036

  Morphological image processing · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title