Systems and methods for categorizing motion event candidates

Patent metadata

Field	Value
Publication number	US-9449229-B1
Application number	US-201514738034-A
Country	US
Kind code	B1
Filing date	Jun 12, 2015
Priority date	Jul 7, 2014
Publication date	Sep 20, 2016
Grant date	Sep 20, 2016

Abstract

The various embodiments described herein include methods, devices, and systems for categorizing motion event candidates. In one aspect, a method includes receiving and processing video frames that include a motion event candidate. The processing includes: (a) obtaining background factors corresponding to a background in at least a subset of the video frames; (b) utilizing the background factors to identify one or more motion entities; (c) for each motion entity, obtaining one or more representative motion vectors based on a motion track of the respective motion entity; (d) identifying one or more features in at least a subset of the video frames; and (e) aggregating the background factors, the representative motion vectors, and the features to generate motion features. The method further includes sending the motion features to an event categorizer, where the event categorizer assigns a motion event category to the motion event candidate based on the received motion features.

First claim

What is claimed is:

1. A method comprising:
at a computing system having one or more processors and memory:
receiving a plurality of video frames, the plurality of video frames including a motion event candidate;
processing the plurality of video frames, the processing comprising:
obtaining one or more background factors corresponding to a background in at least a subset of the plurality of video frames;
utilizing the obtained background factors to identify one or more motion entities in at least a subset of the plurality of video frames;
for each identified motion entity, obtaining one or more representative motion vectors based on a motion track of the respective motion entity;
identifying one or more scene features in at least a subset of the plurality of video frames; and
aggregating the obtained background factors, the obtained representative motion vectors, and the identified scene features to generate a plurality of motion features; and

sending the plurality of motion features to an event categorizer;
wherein the event categorizer assigns a motion event category to the motion event candidate based on the received motion features; and
wherein the motion event category assigned to the motion event candidate is selected from a group consisting of:
one or more known event types;
one or more unknown event types; and
a non-event type.

2. The method of claim 1 , further comprising:
performing object recognition on each identified motion entity; and
classifying each of at least a subset of the one or more motion entities in accordance with the performed object recognition; and
wherein the motion event category is further based on the classified objects.

3. The method of claim 1 , wherein the plurality of video frames correspond to a scene;
the method further comprises obtaining distance information for the scene;
wherein the aggregating includes aggregating the obtained distance information.

4. The method of claim 1 , further comprising:
training the event categorizer, the training comprising:
obtaining a plurality of video clips, each video clip in the plurality of video clips including a respective motion event candidate;
designating a motion event category for each respective motion event candidate;
assigning, via the event categorizer, a motion event category to each respective motion event candidate; and
adjusting the event categorizer based on differences between the assigned motion event categories and the designated motion event categories.

5. The method of claim 1 , wherein the plurality of video frames are associated with a first user;
the method further comprises obtaining user information corresponding to the first user; and
wherein the motion event category is further based on the obtained user information.

6. The method of claim 5 , wherein the user information comprises user feedback corresponding to one or more prior motion event candidates.

7. The method of claim 1 , wherein the plurality of video frames correspond to a scene;
the method further comprises obtaining environmental information corresponding to the scene; and
wherein the motion event category is further based on the obtained environmental information.

8. The method of claim 7 , wherein the environmental information comprises information regarding whether the scene is within a structure.

9. The method of claim 1 , wherein the plurality of video frames correspond to one or more cameras;
the method further comprises obtaining camera information corresponding to the one or more cameras; and
wherein the motion event category is further based on the obtained camera information.

10. The method of claim 9 , wherein the camera information comprises information regarding a relationship between each camera in the one or more cameras and the ground.

11. The method of claim 9 , wherein the camera information comprises camera mode information corresponding to each camera in the one or more cameras.

12. The method of claim 11 , wherein the camera mode information comprises information regarding whether a respective camera is in a low-light mode.

13. The method of claim 1 , further comprising:
receiving a second plurality of video frames, the second plurality of video frames including a second motion event candidate;
processing the second plurality of video frames to generate a second plurality of motion features; and
sending the second plurality of motion features to the event categorizer, wherein the event categorizer assigns a second motion event category to the second motion event candidate based on the second plurality of motion features.

14. The method of claim 13 , wherein processing the second plurality of video frames comprises utilizing information corresponding to the processing of the plurality of video frames.

15. The method of claim 13 , further comprising:
prior to receiving the second plurality of video frames, creating a log entry corresponding to the motion event candidate; and
updating the log entry based on the second motion event category.

16. The method of claim 13 , further comprising utilizing both the plurality of motion features and the second plurality of motion features to assign a motion event category to a third motion event candidate, the third motion event candidate corresponding to a combination of the first motion event candidate and the second motion event candidate.

17. The method of claim 1 , further comprising generating a confidence score for the motion event candidate.

18. A server system comprising:
one or more processors; and
memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a plurality of video frames, the plurality of video frames including a motion event candidate;
processing the plurality of video frames, the processing comprising:
obtaining one or more background factors corresponding to a background in at least a subset of the plurality of video frames;
utilizing the obtained background factors to identify one or more motion entities in at least a subset of the plurality of video frames;
for each identified motion entity, obtaining one or more representative motion vectors based on a motion track of the respective motion entity;
identifying one or more scene features in at least a subset of the plurality of video frames; and
aggregating the obtained background factors, the obtained representative motion vectors, and the identified scene features to generate a plurality of motion features; and

sending the plurality of motion features to an event categorizer;
wherein the event categorizer assigns a motion event category to the motion event candidate based on the received motion features; and
wherein the motion event category assigned to the motion event candidate is selected from a group consisting of:
one or more known event types;
one or more unknown event types; and
a non-event type.

19. The server system of claim 18 , wherein the plurality of video frames correspond to a scene;
wherein the one or more programs further include instructions for obtaining distance information for the scene; and
wherein the aggregating includes aggregating the obtained distance information.

20. The server system of claim 18 , wherein the plurality of video frames are associated with a first user;
wherein the one or more programs further include instructions for obtaining user info

Assignees

• Google Inc

Inventors

• Laska Jason N
• Hua Wei
• Chaudhry Rizwan Ahmed
• Varadharajan Srivatsan
• Heitz Iii George Alban

Classifications

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06F3/0482 Primary

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F18/24

  Classification techniques · CPC title

• G08B13/19669

  Event triggers storage or change of storage policy · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title