Method and system for detecting change of context in video streams

Patent metadata

Field	Value
Publication number	US-10878276-B2
Application number	US-201916415942-A
Country	US
Kind code	B2
Filing date	May 17, 2019
Priority date	Jun 5, 2018
Publication date	Dec 29, 2020
Grant date	Dec 29, 2020

Abstract

Described is a system for detecting change of context in a video stream on an autonomous platform. The system extracts salient patches from image frames in the video stream. Each salient patch is translated to a concept vector. A recurrent neural network is enervated with the concept vector, resulting in activations of the recurrent neural network. The activations are classified, and the classified activations are mapped onto context classes. A change in context class is detected in the image frames, and the system causes the autonomous platform to perform an automatic operation to adapt to the change of context class.

First claim

What is claimed is:

1. A system for detecting change of context in a video stream on an autonomous platform, the system comprising:
one or more processors and a non-transitory memory having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of:
extracting salient patches from a plurality of image frames in the video stream;
translating each salient patch to a concept vector;
enervating a recurrent neural network with the concept vector, resulting in activations of the recurrent neural network;
classifying the activations of the recurrent neural network;
mapping the classified activations onto context classes;
detecting a change of context class in the plurality of image frames; and
causing the autonomous platform to perform an automatic operation to adapt to the change of context class.

2. The system as set forth in claim 1 , where in extracting the salient patches, the one or more processors further perform operations of:
processing each image frame in the plurality of image frames with a feature-generating network, resulting in a set of activations;
applying non-negative matrix factorization to the set of activations in a plurality of layers of the feature-generating network, resulting in potentially salient regions in each image frame; and
performing blob-detection for mapping the potentially salient regions to the salient patches.

3. The system as set forth in claim 1 , where in translating each salient patch to a concept vector, the one or more processors further operations of:
processing each salient patch by a feature-generating network having a plurality of layers;
extracting high-dimensional concept vectors by averaging the set of activations across each layer of the feature-generating network; and
reducing the high-dimensional concept vectors to a low-dimensional concept vector of concept components using Hebbian Principal Component Analysis.

4. The system as set forth in claim 1 , wherein a set of connection weights in the recurrent neural network are a distributed representation of a context.

5. The system as set forth in claim 1 , wherein the recurrent neural network comprises a fully connected layer with spiking leaky integrate-and-fire (LIF) neurons, implementing both excitatory and inhibitory Spike-Timing Dependent Plasticity (STDP) and Short Term Potentiation (STP).

6. The system as set forth in claim 1 , wherein the recurrent neural network comprises a spiking network layer having an excitatory sub-network and inhibitory sub-network, wherein the excitatory sub-network is fully connected and the inhibitory sub-network is randomly connected with a predetermined connection probability, and wherein the excitatory sub-network and the inhibitory sub-network are connected with the predetermined connection probability.

7. A computer implemented method for detecting change of context in a video stream on an autonomous platform, the method comprising an act of:
causing one or more processors to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of:
extracting salient patches from a plurality of image frames in the video stream;
translating each salient patch to a concept vector;
enervating a recurrent neural network with the concept vector, resulting in activations of the recurrent neural network;
classifying the activations of the recurrent neural network;
mapping the classified activations onto context classes;
detecting a change of context class in the plurality of image frames; and
causing the autonomous platform to perform an automatic operation to adapt to the change of context class.

8. The method as set forth in claim 7 , where in extracting the salient patches, the one or more processors further perform operations of:
processing each image frame in the plurality of image frames with a feature-generating network, resulting in a set of activations;
applying non-negative matrix factorization to the set of activations in a plurality of layers of the feature-generating network, resulting in potentially salient regions in each image frame; and
performing blob-detection for mapping the potentially salient regions to the salient patches.

9. The method as set forth in claim 7 , where in translating each salient patch to a concept vector, the one or more processors further operations of:
processing each salient patch by a feature-generating network having a plurality of layers;
extracting high-dimensional concept vectors by averaging the set of activations across each layer of the feature-generating network; and
reducing the high-dimensional concept vectors to a low-dimensional concept vector of concept components using Hebbian Principal Component Analysis.

10. The method as set forth in claim 7 , wherein a set of connection weights in the recurrent neural network are a distributed representation of a context.

11. The method as set forth in claim 7 , wherein the recurrent neural network comprises a fully connected layer with spiking leaky integrate-and-fire (LIF) neurons, implementing both excitatory and inhibitory Spike-Timing Dependent Plasticity (STDP) and Short Term Potentiation (STP).

12. The method as set forth in claim 7 , wherein the recurrent neural network comprises a spiking network layer having an excitatory sub-network and inhibitory sub-network, wherein the excitatory sub-network is fully connected and the inhibitory sub-network is randomly connected with a predetermined connection probability, and wherein the excitatory sub-network and the inhibitory sub-network are connected with the predetermined connection probability.

13. A computer program product for detecting change of context in a video stream on an autonomous platform, the computer program product comprising:
computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors for causing the processor to perform operations of:
extracting salient patches from a plurality of image frames in the video stream;
translating each salient patch to a concept vector;
enervating a recurrent neural network with the concept vector, resulting in activations of the recurrent neural network;
classifying the activations of the recurrent neural network;
mapping the classified activations onto context classes;
detecting a change of context class in the plurality of image frames; and
causing the autonomous platform to perform an automatic operation to adapt to the change of context class.

14. The computer program product as set forth in claim 13 , where in extracting the salient patches, the one or more processors further perform operations of:
processing each image frame in the plurality of image frames with a feature-generating network, resulting in a set of activations;
applying non-negative matrix factorization to the set of activations in a plurality of layers of the feature-generating network, resulting in potentially salient regions in each image frame; and
performing blob-detection for mapping the potentially salient regions to the salient patches.

15. The computer program product as set forth in claim 13 , where in translating each salient patch to a concept vector, the one or more processors further operations of:
processing each salient patch by a feature-generating network having a plurality of layers;
extracting high-dimensional concept vectors by averaging the set of activations across each layer of the feature-generating network; and
reducin

Assignees

• Hrl Lab Llc

Inventors

• Martin Charles E
• Stepp Nigel D
• Kolouri Soheil
• Hoffmann Heiko

Classifications

• G06N3/049 Primary

  Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs · CPC title

• G06V10/768

  using context analysis, e.g. recognition aided by known co-occurring patterns · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title