System and method for coarse-to-fine video object segmentation and re-composition

Patent metadata

Field	Value
Publication number	US-2019164006-A1
Application number	US-201916263961-A
Country	US
Kind code	A1
Filing date	Jan 31, 2019
Priority date	Feb 24, 2016
Publication date	May 30, 2019
Grant date	—

Abstract

Embodiments of the present disclosure include a computer-implemented method that receives a digital image input, the digital image input containing one or more dynamic salient objects arranged over a background. The method also includes performing a tracking operation, the tracking operation identifying the dynamic salient object over one or more frames of the digital image input as the dynamic salient object moves over the background. The method further includes performing a clustering operation, in parallel with the tracking operation, on the digital image input, the clustering operation identifying boundary conditions of the dynamic salient object. Additionally, the method includes combining a first output from the tracking operation and a second output from the clustering operation to generate a third output. The method further includes performing a segmentation operation on the third output, the segmentation operation extracting the dynamic salient object from the digital image input.

First claim

1 . A computer-implemented method, comprising:
receiving a digital video image input, the digital video image input comprising a dynamic salient object arranged over a background; determining a spatial sampling ratio and a temporal sampling ratio; adjusting a spatial resolution and a frame rate according to the determined spatial sampling ratio and the temporal sampling ratio; performing a tracking operation, wherein the tracking operation identifies the dynamic salient object over one or more frames of the digital video image input as the dynamic salient object moves over the background; performing a clustering operation in parallel with the tracking operation on the digital video image input, wherein the clustering operation identifies boundary conditions of the dynamic salient object; combining a first output from the tracking operation and a second output from the clustering operation to generate a third output; and performing a segmentation operation on the third output, wherein the segmentation operation extracts the dynamic salient object from the digital video image input.

2 . The computer-implemented method of claim 1 , wherein determining the spatial sampling ratio and the temporal sampling ratio is based on a percentage of original resolution and frame rate of the digital video image input.

3 . The computer-implemented method of claim 2 , wherein the percentage of original resolution and frame rate of the digital video image input is based on processor computing ability.

4 . The computer-implemented method of claim 1 , wherein the clustering operation comprises supervoxel clustering.

5 . The computer-implemented method of claim 4 , wherein the supervoxel clustering is performed utilizing Simple Linear Interactive Clustering (SLIC).

6 . The computer-implemented method of claim 1 , further comprising applying a bilateral filter to each frame of the digital video image input.

7 . The computer-implemented method of claim 1 , wherein performing the tracking operation comprises point tracking the digital video image input via an algorithm selected from the group consisting of: mean shift tracking, Kanade-Lucas-Tomasi (KLT), tracking learning-detection (TLD), scale-invariant feature transform (SIFT), and Harris corner detection.

8 . The computer-implemented method of claim 1 , further comprising positioning the extracted dynamic salient object onto a different video sequence from the digital video image input to simulate interaction between animated objects and real life objects.

9 . The computer-implemented method of claim 1 , further comprising re-compositioning the extracted dynamic salient object onto a digital image, the digital image being different than frames in the digital video image input.

10 . The computer-implemented method of claim 9 , further comprising applying, to the extracted dynamic salient object, a transformation selected from the group consisting of: a color change, a contrast change, a brightness change, a tone change, a transparency change, and a filter change.

11 . The computer-implemented method of claim 9 , further comprising arranging the extracted dynamic salient object in multiple positions on the digital image to create a moving effect, wherein the digital image is a still photograph.

12 . The computer-implemented method of claim 1 , comprising receiving the digital video image input from a device selected from the group consisting of: a camera, a personal electronic device (PED), a smart phone, a wearable device, and a computer.

13 . The computer-implemented method of claim 12 , wherein the personal electronic device (PED) comprises a processor, a memory, a touchscreen display, and input device, and a network device.

Assignees

• Kodak Alaris Inc

Inventors

• Loui Alexander C
• Zhang Chi

Classifications

• G06T7/215 Primary

  Motion-based segmentation · CPC title

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title