System and method for coarse-to-fine video object segmentation and re-composition

Patent metadata

Field	Value
Publication number	US-10229340-B2
Application number	US-201715441978-A
Country	US
Kind code	B2
Filing date	Feb 24, 2017
Priority date	Feb 24, 2016
Publication date	Mar 12, 2019
Grant date	Mar 12, 2019

Abstract

Embodiments of the present disclosure include a computer-implemented method that receives a digital image input, the digital image input containing one or more dynamic salient objects arranged over a background. The method also includes performing a tracking operation, the tracking operation identifying the dynamic salient object over one or more frames of the digital image input as the dynamic salient object moves over the background. The method further includes performing a clustering operation, in parallel with the tracking operation, on the digital image input, the clustering operation identifying boundary conditions of the dynamic salient object. Additionally, the method includes combining a first output from the tracking operation and a second output from the clustering operation to generate a third output. The method further includes performing a segmentation operation on the third output, the segmentation operation extracting the dynamic salient object from the digital image input.

First claim

The invention claimed is:

1. A computer-implemented method, comprising:
receiving a digital image input, the digital image input containing one or more dynamic salient objects arranged over a background;
performing a tracking operation, the tracking operation identifying the dynamic salient object over one or more frames of the digital image input as the dynamic salient object moves over the background;
performing a clustering operation, in parallel with the tracking operation, on the digital image input, the clustering operation identifying boundary conditions of the dynamic salient object;
combining a first output from the tracking operation and a second output from the clustering operation to generate a third output; and
performing a segmentation operation on the third output, the segmentation operation extracting the dynamic salient object from the digital image input.

2. The computer-implemented method of claim 1 , further comprising creating an effect using an extracted dynamic salient object from the digital image input.

3. The computer-implemented method of claim 1 , wherein the tracking operation comprises a point tracking algorithm and a motion clustering algorithm, performed in series.

4. The computer-implemented method of claim 3 , wherein the point tracking algorithm is performed before the motion clustering algorithm.

5. The computer-implemented method of claim 3 , wherein the point tracking algorithm is a Kanade-Lucas-Tomasi point tracking algorithm and the motion clustering algorithm is a sparse subspace clustering algorithm.

6. The computer-implemented method of claim 1 , wherein the clustering operation comprises supervoxel clustering.

7. The computer-implemented method of claim 1 , further comprising re-compositioning an extracted dynamic salient object onto a digital image, the digital image being different than the digital image input.

8. The computer-implemented method of claim 1 , wherein the segmentation operation comprises a graph-based segmentation, including a coarse segmentation and a fine segmentation, the coarse segmentation being performed before the fine segmentation.

9. A system, comprising:
one or more processors; and
memory including instructions that, when executed by the one or more processors, cause the system to:
receive a digital image input, the digital image input containing one or more dynamic salient objects arranged over a background;
perform a tracking operation, the tracking operation identifying the dynamic salient object over one or more frames of the digital image input as the dynamic salient object moves over the background;
perform a clustering operation, in parallel with the tracking operation, on the digital image input, the clustering operation identifying boundary conditions of the dynamic salient object;
combine a first output from the tracking operation and a second output from the clustering operation to generate a third output; and
perform a segmentation operation on the third output, the segmentation operation extracting the dynamic salient object from the digital image input.

10. The system of claim 9 , where the memory further includes instructions that, when executed by the one or more processors, cause the system to create an effect using an extracted dynamic salient object from the digital image input.

11. The system of claim 9 , wherein the tracking operation comprises a point tracking algorithm and a motion clustering algorithm, performed in series.

12. The system of claim 9 , wherein the clustering operation comprises supervoxel clustering.

13. The system of claim 9 , wherein the segmentation operation comprises a graph-based segmentation including a coarse segmentation and a fine segmentation.

14. The system of claim 9 , wherein the memory further includes instructions that, when executed by the one or more processors, cause the system to re-composition an extracted dynamic salient object onto a digital image, the digital image being different than the digital image input.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause a computing system to:
receive a digital image input, the digital image input containing one or more dynamic salient objects arranged over a background;
perform a tracking operation, the tracking operation identifying the dynamic salient object over one or more frames of the digital image input as the dynamic salient object moves over the background;
perform a clustering operation, in parallel with the tracking operation, on the digital image input, the clustering operation identifying boundary conditions of the dynamic salient object;
combine a first output from the tracking operation and a second output from the clustering operation to generate a third output; and
perform a segmentation operation on the third output, the segmentation operation extracting the dynamic salient object from the digital image input.

16. The non-transitory computer-readable storage medium of claim 15 , further comprising instructions that, when executed by the one or more processors, cause the computing system to create an effect using an extracted dynamic salient object from the digital image input.

17. The non-transitory computer-readable storage medium of claim 15 , wherein the tracking operation comprises a point tracking algorithm and a motion clustering algorithm, performed in series.

18. The non-transitory computer-readable storage medium of claim 15 , wherein the clustering operation comprises supervoxel clustering.

19. The non-transitory computer-readable storage medium of claim 15 , wherein the segmentation operation comprises a graph-based segmentation.

20. The non-transitory computer-readable storage medium of claim 15 , further comprising instructions that, when executed by the one or more processors, cause the computing system to re-composition an extracted dynamic salient object onto a digital image, the digital image being different than the digital image input.

Assignees

• Kodak Alaris Inc

Inventors

• Loui Alexander C
• Zhang Chi

Classifications

• G06T7/215 Primary

  Motion-based segmentation · CPC title

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title