Method for extracting low-rank descriptors from images and videos for querying, classification, and object detection

Patent metadata

Field	Value
Publication number	US-9639761-B2
Application number	US-201414202327-A
Country	US
Kind code	B2
Filing date	Mar 10, 2014
Priority date	Mar 10, 2014
Publication date	May 2, 2017
Grant date	May 2, 2017

Abstract

A method extracts a low-rank descriptor of a video acquired of a scene by first extracting a set of descriptors for each image in the video. The sets of descriptors for the video are aggregated to form a descriptor matrix. Iteratively, a low-rank descriptor matrix is determined from the descriptor matrix, as well as a selection matrix that associates each column in the descriptor matrix to a corresponding column in the low-rank descriptor matrix. The low-rank descriptor matrix is output upon convergence.

First claim

We claim:

1. A method for querying a database of videos, comprising:
extracting a set of descriptors for each image in a video of a scene;
aggregating the sets of descriptors for the video to form a descriptor matrix;
determining a product of a low-rank descriptor matrix and a selection matrix using an orthogonal non-negative matrix factorization of the descriptor matrix, such that each column of the selection matrix has only one nonzero entry and associates a column in the descriptor matrix to a corresponding column in the low-rank descriptor matrix;
wherein the factorization comprises:
determining a low-rank factor by a non-negative least squares minimization; and
determining a selection matrix factor by minimizing a proximal point least squares problem and retaining a largest entry in every column of the selection matrix and setting all other entries to zero;

and
querying the database of videos including at least one video using the low-rank descriptor matrix to produce a search result, wherein the steps are performed in a processor.

2. The method of claim 1 , further comprising:
extracting features from each image in the video; and
aggregating the features to form the set of descriptors for each image in the video.

3. The method of claim 2 , wherein the features are extracted using a scale-invariant feature transform.

4. The method of claim 1 , wherein the set of descriptors for each image in the video are stacked to form a matrix X of size m×N, where m is a length of a feature vector and N is a total number of descriptors extracted from the video.

5. The method of claim 1 , wherein the rank of the low-rank descriptor matrix is less than a length of a feature vector.

6. The method of claim 1 , further comprising:
determining a low-rank class descriptor of each video in the database, wherein each video in the database is associated with a class;
determining a correlation coefficient between the low-rank descriptor matrix and each low-rank class descriptor; and
assigning the class of the video in the database with a largest correlation coefficient to the video of the scene.

7. The method of claim 6 , wherein each video is partitioned into a group of pictures, and the determining and assigning steps are applied to the group of pictures.

8. The method of claim 1 , further comprising:
determining a low-rank class descriptor of each video in the database;
determining a correlation coefficient between the low-rank descriptor matrix and each low-rank class descriptor; and
retrieving the videos in the database with a correlation coefficient larger than a predetermined threshold.

9. The method of claim 1 , wherein the scene includes an object, and further comprising:
subtracting background pixels from each image in the video to obtain a foreground video;
determining a low-rank object descriptor of the foreground video;
determining a low-rank object class descriptor of each video in the database, wherein each video in the database is associated with an object class;
and assigning the object class of the video in the database with a largest correlation coefficient to the foreground video.

10. The method of claim 1 , wherein the scene includes an object, and further comprising:
subtracting background pixels from each image in the video to obtain a foreground video of the scene;
determining a low-rank object descriptor of the foreground video of the scene;
subtracting background pixels from each image of each video in the database to obtain foreground videos in the database;
determining a low-rank object class descriptor of each foreground video in the database, wherein each video in the database is associated with an object class;
and assigning the object class of the video in the database with a largest correlation coefficient to the foreground video of the scene.

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Mansour Hassan
• Rane Shantanu
• Boufounos Petros T
• Vetro Anthony

Classifications

• G06V10/7715 Primary

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06F18/2135

  based on approximation criteria, e.g. principal component analysis · CPC title

• G06F16/75

  Clustering; Classification · CPC title

• G06K9/46

  Physics · mapped topic