System and method for generating semantic annotations

Patent metadata

Field	Value
Publication number	US-9251421-B2
Application number	US-201313918905-A
Country	US
Kind code	B2
Filing date	Jun 15, 2013
Priority date	Sep 13, 2012
Publication date	Feb 2, 2016
Grant date	Feb 2, 2016

Abstract

In accordance with one aspect of the present technique, a method is disclosed. The method includes receiving a new video from one or more sensors and generating a new content graph (CG) based on the new video. The method also includes comparing the new CG with a plurality of prior CGs. The method further includes identifying a first portion of the new CG matching a portion of a first prior CG and a second portion of the new CG matching a portion of the second prior CG. The method further includes analyzing a first set of semantic annotations (SAs) associated with the portion of the first prior CG and a second set of SAs associated with the portion of the second prior CG. The method further includes generating a sequence of SAs for the new video based on the analysis of the first and the second set of SAs.

First claim

The invention claimed is:

1. A method, comprising:
receiving a new video from one or more sensors;
generating a new content graph (CG) based on the new video;
comparing the new CG with a plurality of prior CGs, wherein the plurality of prior CGs are generated from a plurality of previously received videos;
identifying a first portion of the new CG matching a portion of a first prior CG among the plurality of prior CGs and a second portion of the new CG matching a portion of a second prior CG among the plurality of prior CGs;
analyzing a first set of semantic annotations (SAs) associated with the portion of the first prior CG and a second set of SAs associated with the portion of the second prior CG; and
generating a sequence of SAs that temporally corresponds with the new video by combining the first set of SAs and the second set of SAs based on the analysis of the first and the second set of SAs.

2. The method of claim 1 , wherein the new CG includes a plurality of nodes interconnected via one or more edges.

3. The method of claim 2 , wherein the plurality of nodes represent at least one of a plurality of objects and a plurality of activities in the new video.

4. The method of claim 2 , wherein the one or more one or more edges represent at least one of a spatial relationship, a temporal relationship, and a dynamic relationship between the plurality of nodes.

5. The method of claim 2 , wherein identifying the portion of the first prior CG further comprises determining at least one of a number of matching nodes and a number of matching edges between the first portion of the new CG and the portion of the first prior CG.

6. The method of claim 1 , wherein analyzing the first and the second set of SAs further comprises analyzing statistical data associated with the first and the second set of SAs.

7. The method of claim 1 , further comprising:
determining whether the sequence of SAs satisfies a notification category; and
sending a notification in response to determining that the sequence of SAs satisfies the notification category.

8. The method of claim 7 , wherein the notification category includes at least one of safety, theft, vandalism, and business opportunity.

9. A system, comprising:
at least one processor;
a graph module stored in a memory and executable by the at least one processor, the graph module configured for receiving a new video from one or more sensors and generating a new content graph (CG) based on the new video;
a comparison module stored in the memory and executable by the at least one processor, the comparison module communicatively coupled to the graph module for comparing the new CG with a plurality of prior CGs and identifying a first portion of the new CG matching a portion of a first prior CG among the plurality of prior CGs and a second portion of the new CG matching a portion of a second prior CG among the plurality of prior CGs, wherein the plurality of prior CGs are generated from a plurality of previously received videos; and
a narrative module stored in the memory and executable by the at least one processor, the narrative module communicatively coupled to the comparison module for analyzing a first set of semantic annotations (SAs) associated with the portion of the first prior CG and a second set of SAs associated with the portion of the second prior CG and generating a sequence of SAs that temporally corresponds with the new video by combining the first set of SAs and the second set of SAs based on the analysis of the first and the second set of SAs.

10. The system of claim 9 , wherein the new CG includes a plurality of nodes interconnected via one or more edges.

11. The system of claim 10 , wherein the comparison module is further configured to determine at least one of a number of matching nodes and a number of matching edges between the first portion of the new CG and the portion of the first prior CG.

12. The system of claim 9 , wherein the narrative module is further configured to analyze statistical data associated with the first and the second set of SAs.

13. The system of claim 9 , further comprising a notification module for determining whether the sequence of SAs satisfies a notification category and sending a notification to a user in response to determining that the sequence of SAs satisfies the notification category.

14. A computer program product comprising a non-transitory computer readable medium encoded with instructions that, in response to execution by at least one processor, cause the processor to perform operations comprising:
receiving a new video from one or more sensors;
generating a new content graph (CG) based on the new video;
comparing the new CG with a plurality of prior CGs, wherein the plurality of prior CGs are generated from a plurality of previously received videos;
identifying a first portion of the new CG matching a portion of a first prior CG among the plurality of prior CGs and a second portion of the new CG matching a portion of a second prior CG among the plurality of prior CGs;
analyzing a first set of semantic annotations (SAs) associated with the portion of the first prior CG and a second set of SAs associated with the portion of the second prior CG; and
generating a sequence of SAs that temporally corresponds with the new video by combining the first set of SAs and the second set of SAs based on the analysis of the first and the second set of SAs.

15. The computer program product of claim 14 , wherein the new CG includes a plurality of nodes interconnected via one or more edges.

16. The computer program product of claim 15 , wherein the plurality of nodes represent at least one of a plurality of objects and a plurality of activities in the new video.

17. The computer program product of claim 15 , wherein the one or more one or more edges represent at least one of a spatial relationship, a temporal relationship, and a dynamic relationship between the plurality of nodes.

18. The computer program product of claim 14 , further causing the processor to perform operations comprising determining at least one of a number of matching nodes and a number of matching edges between the first portion of the new CG and the portion of the first prior CG.

19. The computer program product of claim 14 , further causing the processor to perform operations comprising analyzing statistical data associated with the first and the second set of SAs.

20. The computer program product of claim 14 , further causing the processor to perform operations comprising:
determining whether the sequence of SAs satisfies a notification category;
sending a notification to a user in response to determining that the sequence of SAs satisfies the notification category.

Assignees

• Gen Electric

Inventors

• Tu Peter Henry
• Rittscher Jens
• Guan Li

Classifications

• G06V20/70 Primary

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• G06V10/7635

  based on graphs, e.g. graph cuts or spectral clustering · CPC title

• G06V10/70

  using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title