System and method for video classification using a hybrid unsupervised and supervised multi-layer architecture

Patent metadata

Field	Value
Publication number	US-9946933-B2
Application number	US-201615240561-A
Country	US
Kind code	B2
Filing date	Aug 18, 2016
Priority date	Aug 18, 2016
Publication date	Apr 17, 2018
Grant date	Apr 17, 2018

Abstract

A computer-implemented video classification method and system are disclosed. The method includes receiving an input video including a sequence of frames. At least one transformation of the input video is generated, each transformation including a sequence of frames. For the input video and each transformation, local descriptors are extracted from the respective sequence of frames. The local descriptors of the input video and each transformation are aggregated to form an aggregated feature vector with a first set of processing layers learned using unsupervised learning. An output classification value is generated for the input video, based on the aggregated feature vector with a second set of processing layers learned using supervised learning.

First claim

The invention claimed is:

1. A video classification method comprising:
with at least one processor of one or more computing devices:
receiving an input video comprising a sequence of frames;
generating at least one transformation of the input video, each transformation comprising a sequence of frames;
for the input video and each transformation, extracting local descriptors from the respective sequence of frames;
aggregating the local descriptors of the input video and each transformation to form an aggregated feature vector with a first set of processing layers learned using unsupervised learning; and
generating an output classification value for the input video based on the aggregated feature vector with a second set of processing layers learned using supervised learning.

2. The method of claim 1 , wherein the second set of processing layers comprises a neural network.

3. The method of claim 2 , wherein at least one the second set of processing layers performs a linear vector projection and a non-linear vector transformation.

4. The method of claim 1 , wherein the at least one transformation comprises a plurality of transformations.

5. The method of claim 1 , wherein the at least one transformation comprises at least one of:
repeating frames of the input video,
skipping frames of the input video,
color modifications to the input video;
affine transformations of frames of the input video; and
projective transformations of frames of the input video.

6. The method of claim 5 , wherein the at least transformation comprises at least two of:
repeating frames of the input video,
skipping frames of the input video,
color modifications to the input video;
affine transformations of frames of the input video; and
projective transformations of frames of the input video.

7. The method of claim 1 , wherein the extracting of local descriptors comprises:
sampling the local descriptors along an optical flow trajectory identified from the plurality of frames.

8. The method of claim 1 , wherein the generating of the aggregated feature vector comprises, for each of the input video and each transformation, generating a respective feature vector based on the respective local descriptors and aggregating the feature vectors.

9. The method of claim 1 , wherein the extracting local descriptors comprises:
extracting a plurality of spatio-temporal features from the at least one transformation of the input video;
the aggregating of the local descriptors comprising:
stacking the extracted spatio-temporal features into a matrix;
encoding the matrix;
pooling the encodings of the matrix to generate an encoding vector; and
normalizing the encoding vector to obtain a single aggregated feature vector.

10. The method of claim 1 , wherein the aggregating of the local descriptors comprises:
encoding the local descriptors extracted from each transformation using a generative model; and;
aggregating the encoded local descriptors.

11. The method of claim 10 , wherein the encoding of the local descriptors comprises encoding the local descriptors extracted from the sequence of frames using a Fisher vector encoding.

12. The method of claim 1 , further comprising:
with one of the at least one processors, projecting the aggregating feature vector into a space of lower dimensionality than the dimensionality of the aggregating feature vector to produce an aggregated feature vector of lower dimensionality prior to generating the output classification value.

13. The method of claim 1 , further comprising, with one of the at least one processors, training the second sequence of layers comprising:
for each training video of a set of training videos, generating a training video aggregated feature vector representing the training video with the first set of processing layers; and
training the second sequence of layers on a training set comprising the generated training video feature vectors annotated by labels of the represented training videos.

14. The method of claim 13 , wherein the first layers are trained without using the labels of the set of training videos.

15. The method of claim 1 , further comprising, with one of the at least one processors, outputting at least one of the classification value and information generated therefrom.

16. The method of claim 1 , wherein the classification value comprises a vector of classification values.

17. A computer program product comprising a non-transitory recording medium storing instructions, which when executed on a computer, causes the computer to perform a method comprising:
receiving an input video comprising a sequence of frames;
generating at least one transformation of the input video, each transformation comprising a sequence of frames;
for the input video and each transformation, extracting local descriptors from the respective sequence of frames;
aggregating the local descriptors of the input video and each transformation to form an aggregated feature vector with a first set of processing layers learned using unsupervised learning; and
generating an output classification value for the input video based on the aggregated feature vector with a second set of processing layers learned using supervised learning.

18. A system comprising memory which stores instructions for performing a method and a processor in communication with the memory for executing the instructions, the method comprising:
receiving an input video comprising a sequence of frames;
generating at least one transformation of the input video, each transformation comprising a sequence of frames;
for the input video and each transformation, extracting local descriptors from the respective sequence of frames;
aggregating the local descriptors of the input video and each transformation to form an aggregated feature vector with a first set of processing layers learned using unsupervised learning; and
generating an output classification value for the input video based on the aggregated feature vector with a second set of processing layers learned using supervised learning.

19. A video classification system comprising:
a transformation generator which generates at least one transformation of an input video comprising a sequence of frames, each transformation comprising a sequence of frames;
a feature vector generator which, for the input video and each transformation, extracts local descriptors from the respective sequence of frames and aggregates the local descriptors of the input video and each transformation to form an aggregated feature vector with a first set of processing layers learned using unsupervised learning; and
a classifier component which generates an output classification value for the input video based on the aggregated feature vector with a second set of processing layers learned using supervised learning; and
a hardware processor which implements the transformation generator, feature vector generator and classifier component.

20. The system of claim 19 , further comprising an output component, implemented by the hardware processor, which outputs at least one of the classification value and information based thereon.

21. The system of claim 19 , wherein the at least one transformation comprises a plurality of transformations.

22. A method for classifying a video, the method comprising:
with at least one processor of one or more computing devices:
receiving an input video;
generating a plurality of transformations of the input video

Assignees

• Xerox Corp

Inventors

• De Souza César Roberto
• Gaidon Adrien
• Vig Eleonora
• Lopez Antonio M

Classifications

• G06V10/7753

  Incorporation of unlabelled data, e.g. multiple instance learning [MIL] · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06V10/32 Primary

  Normalisation of the pattern dimensions · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title