Activity recognition systems and methods

Patent metadata

Field	Value
Publication number	US-10216984-B2
Application number	US-201815875681-A
Country	US
Kind code	B2
Filing date	Jan 19, 2018
Priority date	Jun 17, 2014
Publication date	Feb 26, 2019
Grant date	Feb 26, 2019

Abstract

An activity recognition system is disclosed. A plurality of temporal features is generated from a digital representation of an observed activity using a feature detection algorithm. An observed activity graph comprising one or more clusters of temporal features generated from the digital representation is established, wherein each one of the one or more clusters of temporal features defines a node of the observed activity graph. At least one contextually relevant scoring technique is selected from similarity scoring techniques for known activity graphs, the at least one contextually relevant scoring technique being associated with activity ingestion metadata that satisfies device context criteria defined based on device contextual attributes of the digital representation, and a similarity activity score is calculated for the observed activity graph as a function of the at least one contextually relevant scoring technique, the similarity activity score being relative to at least one known activity graph.

First claim

What is claimed is:

1. An activity recognition apparatus comprising:
at least one sensor configured to generate a digital representation of an environment of activity according to one or more sensing modalities;
a memory storing known activity data objects, wherein each known activity data object relates to a known activity and includes similarity scoring techniques and clustered features; and
an activity recognition device electronically coupled with the sensor and the memory and having a processor, wherein, upon execution of software instructions stored on a non-transitory computer readable medium, the processor is configured to:
generate a plurality of features from the digital representation using at least one feature detection algorithm;
establish an observed activity data object comprising one or more observed feature clusters generated from the plurality of features;
calculate a similarity activity score for the observed activity data object relative to at least one of the known activity data objects as a function of the similarity scoring techniques that are contextually relevant to the environment, the clustered features, and the observed feature clusters;
access an activity recognition results set as a function of the similarity activity score; and
initiate an action regarding the environment based on the activity recognition results set.

2. The apparatus of claim 1 , wherein the known activity includes at least one of a body movement and an interaction.

3. The apparatus of claim 1 , wherein the known activity data objects comprise at least a part of a template for interactions.

4. The apparatus of claim 1 , wherein the activity recognition device is further configured to:
convert aspects of the digital representation to an observed activity graph; and compare the observed activity graph to known activity graphs.

5. The apparatus of claim 1 , wherein the one or more sensing modalities comprise at least one of image data, video data, tactile data, kinesthetic data, temperature data, kinematic data, 3D registration data, and radio signal or wireless data.

6. The apparatus of claim 5 , wherein the image data comprises at least one of ultrasound, infrared, visible spectrum data.

7. The apparatus of claim 1 , wherein the digital representation comprises one or more of video data, still image data, audio data, and accelerometer data.

8. The apparatus of claim 7 , wherein the digital representation comprises a video of a procedure.

9. The apparatus of claim 8 , wherein contextual relevance relates to one or more of when the procedure is performed, information about the procedure, a provider associated with the procedure, and a location of the procedure.

10. The apparatus of claim 1 , wherein the at least one feature detection algorithm includes one of the following: a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG), and Transform Invariant Low-rank Textures (TILT) feature detection algorithm.

11. The apparatus of claim 1 , wherein the at least one sensor device is further configured to observe the environment over a time period or within a time frame.

12. The apparatus of claim 11 , wherein at least some of the plurality of features describe a temporal or spatial relationship among comparable events in time.

13. The apparatus of claim 1 , wherein the activity recognition device is further configured to determine contextual relevance based on ingestion metadata.

14. The apparatus of claim 13 , wherein the activity recognition device is further configured to select the ingestion metadata used to determine contextual relevance based on one or more domain-specific attributes.

15. The apparatus of claim 13 , wherein the ingestion metadata conforms to a defined attribute namespace.

16. The apparatus of claim 1 , wherein the activity recognition device is further configured to:
recognize one or more objects in the digital representation using at least some of the plurality of features; and
retrieve object information related to the one or more recognized objects.

17. The apparatus of claim 16 , wherein the activity recognition device is further configured to use the object information to determine contextual relevance.

18. The apparatus of claim 1 , wherein the similarity scoring techniques include at least one of a Euclidean distance, linear kernel, polynomial kernel, Chi-squared kernel, Cauchy kernel, histogram intersection kernel, Hellinger's kernel, Jensen-Shannon kernel, hyperbolic tangent (sigmoid) kernel, rational quadratic kernel, multiquadratic kernel, inverse multiquadratic kernel, circular kernel, spherical kernel, wave kernel, power kernel, log kernel, spline kernel, Bessel kernel, generalized T-Student kernel, Bayesian kernel, wavelet kernel, radial basis function (RBF), exponential kernel, Laplacian kernel, ANOVA kernel and B-spline kernel function.

19. The apparatus of claim 1 , wherein the activity recognition device is further configured to select the similarity scoring techniques according to a data modality.

20. The apparatus of claim 1 , wherein the similarity scoring techniques reflect a relative confidence of data from each of a plurality of sensing modalities.

21. The apparatus of claim 1 , wherein the activity recognition results set comprises at least one of an activity identifier, a search result, a classification, a recommendation, an anomaly, a warning, a segmentation, a command, a ranking, context relevant information, content information, and an action prediction.

22. The apparatus of claim 21 , wherein the action prediction is based on variations of known activities.

23. The apparatus of claim 1 , wherein initiating the action comprises executing a command.

24. The apparatus of claim 1 , wherein initiating the action comprises generating an alert.

25. The apparatus of claim 1 , wherein the environment comprises a volumetric space.

26. The apparatus of claim 1 , wherein the activity recognition device comprises a surveillance system.

Assignees

• Nant Holdings Ip Llc

Inventors

• Wnuk Kamil
• Witchey Nicholas J

Classifications

• G06F16/7837

  using objects detected or recognised in the video content · CPC title

• G06F16/285

  Clustering or classification · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• G06F16/2228

  Indexing structures · CPC title

• G06F16/23

  Updating · CPC title