Graph-based framework for video object segmentation and extraction in feature space

Patent metadata

Field	Value
Publication number	US-10192117-B2
Application number	US-201615167327-A
Country	US
Kind code	B2
Filing date	May 27, 2016
Priority date	Jun 25, 2015
Publication date	Jan 29, 2019
Grant date	Jan 29, 2019

Abstract

A method for graph-based spatiotemporal video segmentation and automatic target object extraction in high-dimensional feature space includes using a processor to automatically analyze an entire volumetric video sequence; using the processor to construct a high-dimensional feature space that includes color, motion, time, and location information so that pixels in the entire volumetric video sequence are reorganized according to their unique and distinguishable feature vectors; using the processor to create a graph model that fuses the appearance, spatial, and temporal information of all pixels of the video sequence in the high-dimensional feature space; and using the processor to group pixels in the graph model that are inherently similar and assign the same labels to them to form semantic spatiotemporal key segments.

First claim

We claim:

1. A method for graph-based spatiotemporal video segmentation and automatic target object extraction in high-dimensional feature space, comprising:
a) automatically analyzing an entire volumetric video sequence;
b) constructing a high-dimensional feature space that includes color, motion, time, and location information so that pixels in the entire volumetric video sequence are reorganized according to their unique and distinguishable feature vectors;
c) creating a graph model that fuses appearance, spatial, and temporal information of all pixels of the video sequence in the high-dimensional feature space, wherein the graph model represents each pixel as a graph node, and two pixels are connected by an edge based on similarity criteria;
d) grouping pixels in the graph model that are inherently similar and assign the same labels to them to form semantic spatiotemporal key segments; and
e) using the semantic spatiotemporal key segments as an input to an initial background/foreground model combined with a graph cut algorithm to label at least one target object.

2. The method of claim 1 wherein the graph cut algorithm is used to analyze region level volumetric segments for each segment obtained from the previous video segmentation stage to create nodes and using edges between nodes to reflect their mutual similarity considering both spatial and temporal coherence.

3. The method of claim 1 wherein intra-cluster connectivity is used to correct spatial and temporal inconsistency due to sudden motion changes or occlusion.

4. The method of claim 1 wherein the high-dimensional feature space is a seven dimension feature space.

5. The method of claim 1 wherein a k-nearest neighbor search is used in step d) to group pixels that are inherently similar and assign the same labels to them.

Assignees

• Kodak Alaris Inc

Inventors

• Loui Alexander C
• Fan Lei

Classifications

• G06V10/7635

  based on graphs, e.g. graph cuts or spectral clustering · CPC title

• G06V20/49 Primary

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06F18/2323

  based on graph theory, e.g. minimum spanning trees [MST] or graph cuts · CPC title

• G06K9/00765

  Physics · mapped topic