Activity recognition systems and methods

Patent metadata

Field	Value
Publication number	US-9547678-B2
Application number	US-201514741830-A
Country	US
Kind code	B2
Filing date	Jun 17, 2015
Priority date	Jun 17, 2014
Publication date	Jan 17, 2017
Grant date	Jan 17, 2017

Abstract

An activity recognition system is disclosed. A plurality of temporal features is generated from a digital representation of an observed activity using a feature detection algorithm. An observed activity graph comprising one or more clusters of temporal features generated from the digital representation is established, wherein each one of the one or more clusters of temporal features defines a node of the observed activity graph. At least one contextually relevant scoring technique is selected from similarity scoring techniques for known activity graphs, the at least one contextually relevant scoring technique being associated with activity ingestion metadata that satisfies device context criteria defined based on device contextual attributes of the digital representation, and a similarity activity score is calculated for the observed activity graph as a function of the at least one contextually relevant scoring technique, the similarity activity score being relative to at least one known activity graph.

First claim

What is claimed is:

1. An activity recognition system comprising:
an activity database configured to store similarity scoring techniques for known activity graphs, each of the similarity scoring techniques being associated with activity ingestion metadata; and
an activity recognition device coupled with the activity database and configured to:
generate a plurality of temporal features from a digital representation of an observed activity using a feature detection algorithm;
establish an observed activity graph comprising one or more clusters of temporal features generated from the digital representation, wherein each one of the one or more clusters of temporal features defines a node of the observed activity graph;
select at least one contextually relevant scoring technique from the similarity scoring techniques for known activity graphs, the at least one contextually relevant scoring technique being associated with activity ingestion metadata that satisfies device context criteria defined based on device contextual attributes of the digital representation; and
calculate a similarity activity score for the observed activity graph as a function of the at least one contextually relevant scoring technique, the similarity activity score being relative to at least one known activity graph.

2. The system of claim 1 , wherein the activity database is further configured to store the known activity graphs.

3. The system of claim 2 , wherein the activity recognition device is further configured to calculate the similarity activity score as a function of nodes within the known activity graphs and nodes within the observed activity graph.

4. The system of claim 1 , wherein the activity database is stored, at least partially, in a memory of the activity recognition device.

5. The system of claim 1 , wherein the activity recognition device comprises at least one of a tablet computer, personal computer, camera, cell phone, kiosk, appliance, vehicle, robot, and game device.

6. The system of claim 1 , wherein the device contextual attributes and the activity ingestion metadata adhere to a common namespace.

7. The system of claim 1 , wherein the activity ingestion metadata comprises domain-specific attributes.

8. The system of claim 1 , wherein the activity ingestion metadata comprises object attributes.

9. The system of claim 1 , wherein the activity ingestion metadata comprises at least one of location attributes, time attributes, position attributes and orientation attributes.

10. The system of claim 1 , wherein the known activity graphs and the observed activity graph comprise directed acyclic graphs.

11. The system of claim 1 , wherein the nodes of the observed activity graph comprise clusters of feature trajectories.

12. The system of claim 1 , wherein the digital representation comprises one or more of image data, video data, audio data, tactile data, kinesthetic data, temperature data, kinematic data and radio signal data.

13. The system of claim 1 , wherein the feature detection algorithm includes at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm.

14. The system of claim 1 , wherein the known activity graphs comprise activity graph primitives.

15. The system of claim 14 , wherein the activity graph primitives include at least one of an object persistence, object transformation, object-to-object conserved interaction, object segmentation, object creation, object destruction and object NULL interaction.

16. The system of claim 1 , wherein the known activity graphs include nodes that comprise other known activity graphs.

17. The system of claim 1 , wherein the known activity graphs comprise key frames.

18. The system of claim 1 , wherein the activity recognition device is further configured to generate an activity recognition result set based on the similarity activity score.

19. The system of claim 18 , wherein the activity recognition device is further configured to assign a classification to the observed activity graph, wherein the classification comprises the activity recognition result set.

20. The system of claim 18 , wherein the activity recognition result set comprises an activity prediction with respect to the observed activity graph.

21. The system of claim 18 , wherein the activity recognition result set comprises at least one of an address, an activity identifier, a search result, a recommendation, an anomaly, a warning, a segmentation, a command, a ranking, context relevant information, content information, a promotion and an advertisement.

22. The system of claim 1 , wherein the activity recognition device is further configured to establish a mapping of a static image from the digital representation into a graph space of at least one of the known activity graphs by mapping image features to nodes of the at least one of the known activity graphs.

23. The system of claim 22 , wherein the activity recognition device is further configured to generate an action prediction based on the mapping and the nodes of the at least one of the known activity graphs.

24. The system of claim 1 , wherein the similarity scoring techniques include at least one of a Euclidean distance, linear kernel, polynomial kernel, Chi-squared kernel, Cauchy kernel, histogram intersection kernel, Hellinger's kernel, Jensen-Shannon kernel, hyperbolic tangent (sigmoid) kernel, rational quadratic kernel, multiquadratic kernel, inverse multiquadratic kernel, circular kernel, spherical kernel, wave kernel, power kernel, log kernel, spline kernel, Bessel kernel, generalized T-Student kernel, Bayesian kernel, wavelet kernel, radial basis function (RBF), exponential kernel, Laplacian kernel, ANOVA kernel and B-spline kernel function.

25. The system of claim 1 , wherein the similarity scoring techniques comprise at least one set of node context-based weights.

26. The system of claim 25 , wherein the at least one set of node context-based weights comprises a matrix of node weights.

27. A method of activity recognition at an activity recognition device, the method comprising:
generating a plurality of temporal features from a digital representation of an observed activity using a feature detection algorithm;
establishing an observed activity graph comprising one or more clusters of temporal features generated from the digital representation, wherein each one of the one or more clusters of temporal features defines a node of the observed activity graph;
selecting at least one contextually relevant scoring technique from similarity scoring techniques for known activity graphs, the at least one contextually relevant scoring technique being associated with activity ingestion metadata that satisfies device context criteria defined based on device contextual attributes of the digital representation; and
calculating a similarity activity score for the observed activity graph as a function of the at least one contextually relevant scoring technique, the similarity activity score being relative to at least one known activit

Assignees

• Nant Holdings Ip Llc

Inventors

• Wnuk Kamil
• Witchey Nicholas J

Classifications

• B25J9/1697

  Vision controlled systems · CPC title

• G06F16/24578

  using ranking · CPC title

• G06F16/7837

  using objects detected or recognised in the video content · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title