System and method for activity recognition

Patent metadata

Field	Value
Publication number	US-9278255-B2
Application number	US-201213723141-A
Country	US
Kind code	B2
Filing date	Dec 20, 2012
Priority date	Dec 9, 2012
Publication date	Mar 8, 2016
Grant date	Mar 8, 2016

Abstract

A method for automatic recognition of human activity is provided and includes the steps of decomposing human activity into a plurality of fundamental component attributes needed to perform an activity and defining ontologies of fundamental component attributes from the plurality of the fundamental component attributes identified during the decomposing step for each of a plurality of different targeted activities. The method also includes the steps of converting a data stream captured during a performance of an activity performed by a human into a sequence of fundamental component attributes and classifying the performed activity as one of the plurality of different targeted activities based on a closest match of the sequence of fundamental component attributes obtained during the converting step to at least a part of one of the ontologies of fundamental component attributes defined during the defining step. A system for performing the method is also disclosed.

First claim

We claim:

1. A method for automatic recognition of human activity, comprising the steps of:
decomposing a human activity into a plurality of fundamental component attributes needed to perform the human activity, wherein the human activity is included in a training set of activities;
defining ontologies of fundamental component attributes from the plurality of the fundamental component attributes identified during said decomposing step for each of a plurality of different targeted activities;
converting a data stream, the data stream captured during a performance by a human of a performed activity, into a sequence of fundamental component attributes; and
classifying the performed activity as one of the plurality of different targeted activities based on a closest match of the sequence of fundamental component attributes obtained during said converting step to at least a part of one of the ontologies of fundamental component attributes defined during said defining step, wherein the performed activity is not included in the training set of activities, the classifying comprising selecting only unseen classes in an attribute space;
wherein each of the fundamental component attributes is defined from a sequence of features, and further comprising the step of extracting features from the data stream with computations in at least one of time domain and frequency domain; and
wherein the data stream provides a time-sequence of features, and wherein, during said classifying step, a feature at each time slice of the data stream is compared to features of the fundamental component attributes at a corresponding time slice within the ontologies to determine a closest match.

2. The method according to claim 1 , wherein the plurality of fundamental component attributes include attributes corresponding to at least one of a body motion and a body position.

3. The method according to claim 1 , wherein the data stream is an electronic data stream including data captured by at least one motion sensor worn on the human during the performance of the performed activity.

4. The method according to claim 1 , wherein the data stream is video data of the human during the performance of the performed activity.

5. The method according to claim 1 , wherein said step of defining ontologies includes entry of a combination of fundamental component attributes in a database for each of the plurality of different targeted activities.

6. The method according to claim 1 , wherein each of the different targeted activities includes a unique sequence and combination of fundamental component attributes.

7. A method of automatically recognizing a physical activity being performed by a human, comprising the steps of:
electronically decomposing training data obtained for each of a plurality of different physical activities within a training set of physical activities into a plurality of component attributes needed to perform the physical activities within the training set;
defining ontologies of component attributes from the plurality of component attributes identified during said decomposing step for each of a plurality of different physical activities within a targeted set of different physical activities, the targeted set being different from the training set, wherein the plurality of different physical activities within the targeted set includes at least one physical activity not included within the training set;
electronically capturing a data stream representing an actual physical activity performed;
electronically converting the data stream obtained during said capturing step into a plurality of component attributes; and
automatically classifying the actual physical activity being performed by comparing the plurality of component attributes obtained during said converting step to one of the ontologies of component attributes defined during said defining step; wherein the automatically classifying further comprises classifying the actual physical activity for at least one class of physical activity wherein at least one of the component attributes has not been included within the training set, the classifying comprising selecting only unseen classes in an attribute space;
wherein each of the component attributes is defined from a sequence of features, and further comprising the step of extracting features from the data stream with computations in at least one of time domain and frequency domain; and
wherein the data stream provides a time-sequence of features, and wherein, during said classifying step, a feature at each time slice of the data stream is compared to features of the component attributes at a corresponding time slice within the ontologies to determine a closest match.

8. The method according to claim 7 , wherein the plurality of different physical activities within the training set is a subset of the plurality of different physical activities within the targeted set.

9. The method according to claim 7 , wherein the data stream includes data captured by at least one input source selected from a group consisting of a motion sensor, a video recorder, a sound recording device, a location sensor, a temperature sensor, a pressure sensor, an ambient light sensor, a heartrate sensor, and a proximity sensor.

10. The method according to claim 7 , wherein the features are extracted by computations in at least one of time domain and frequency domain, and wherein the computations include at least one of mean, standard deviation, pair-wise correlation, cross correlation, slope, zero crossing rate, Fourier transform coefficients, and spectral flux.

11. The method according to claim 7 , further comprising the steps of:
determining if there is a match of the features extracted from the data stream obtained during said capturing step to features produced by the training data for one of the plurality of different physical activities within the training set;
if there is a match to one of the plurality of different physical activities within the training set, a matched physical activity within the training set is recognized as the actual physical activity; and
if there is not a match to one of the plurality of different human physical activities within the training set, said converting and classifying steps are performed to predict a physical activity from the targeted set.

12. The method according to claim 7 , wherein said step of defining ontologies includes entry of a combination of component attributes in a database for each physical activity without training data.

13. The method according to claim 7 , wherein the component attributes include attributes corresponding to one of a body motion and body position, and wherein each of the different physical activities is composed of a unique sequence and combination of component attributes.

14. The method according to claim 7 , wherein a sequence and combination of the different physical activities corresponds to a higher level activity, and further comprising the step of recognizing the higher level activity based on the sequence and combination of physical activities classified during said classifying step.

15. A system for automatically recognizing physical activity of a human, comprising:
a feature extractor configured to receive electronic input data captured by a sensor relative to a physical activity and to identify features from the input data; and
an attribute-based activity recognizer having:
an attribute detector for electronically determining an attribute as defined by a sequence of features, and
an activity classifier for classifying and outputting a prediction of the physical activity based on a

Assignees

• Gen Instrument Corp
• Univ Carnegie Mellon
• Arris Entpr Inc

Inventors

• Cheng Heng-Tze
• Davis Paul C
• Li Jianguo
• You Di

Classifications

• G06F2218/00

  Aspects of pattern recognition specially adapted for signal processing · CPC title

• A63B24/00 Primary

  Electric or electronic controls for exercising apparatus of preceding groups; {Controlling or monitoring of exercises, sportive games, training or athletic performances} · CPC title

• G06K9/00342

  Physics · mapped topic

• G06K9/00

  Physics · mapped topic

• G06F18/00 Primary

  Pattern recognition · CPC title