Apparatus and method for video sensor-based human activity and facial expression modeling and recognition

Patent metadata

Field	Value
Publication number	US-9489568-B2
Application number	US-201414307342-A
Country	US
Kind code	B2
Filing date	Jun 17, 2014
Priority date	Jun 5, 2009
Publication date	Nov 8, 2016
Grant date	Nov 8, 2016

Abstract

An apparatus and method for human activity and facial expression modeling and recognition are based on feature extraction techniques from time sequential images. The human activity modeling includes determining principal components of depth and/or binary shape images of human activities extracted from video clips. Independent Component Analysis (ICA) representations are determined based on the principal components. Features are determined through Linear Discriminant Analysis (LDA) based on the ICA representations. A codebook is determined using vector quantization, Observation symbol sequences in the video clips am determined. And human activities are learned using the Hidden Markov Model (HMM) based on status transition and an observation matrix.

First claim

What is claimed is:

1. A human activity recognition method comprising:
determining, by a controller, principal components of depth images and binary shape images of human activities extracted from a video clip;
performing, by the controller, a human activity modeling based on the principal components; and
determining, by the controller, a human activity model which maximizes a likelihood probability among activity models in a dictionary according to an output value of the human activity modeling,
wherein the depth images of the human activities are different images than the binary shape images of the human activities, and the video clip comprises successive images that are based on a video sensor.

2. The human activity recognition method of claim 1 , wherein determining the principal components of depth images and binary shape images of human activities extracted from the video clip comprises:
determining at least one principal component representing an entire image based on a human body representation of the depth images and binary shape images.

3. The human activity recognition method of claim 1 , wherein the video clip is represented using a Hidden Markov Model (HMM) based on a status transition and a symbol observation matrix.

4. The human activity recognition method of claim 1 , wherein determining the human activity model which maximizes the likelihood probability among the activity models in the dictionary according to an output value of the human activity modeling comprises:
determining the human activity model which maximizes the likelihood probability from a human activity HMM dictionary.

5. The human activity recognition method of claim 1 , wherein performing the human activity modeling based on the principal components comprises;
determining Independent Component Analysis (ICA) representations based on the principal components;
determining features through Linear Discriminant Analysis (LDA) based on the ICA representations;
determining a codebook using vector quantization;
determining observation symbol sequences in the video clip; and
identifying human activities using a HMM based on a status transition and an observation matrix.

6. The human activity recognition method of claim 5 , wherein determining ICA representations based on the principal components comprises:
determining independent ICA representations in the principal components.

7. The human activity recognition method of claim 5 , wherein determining features through LDA based on the ICA representations comprises:
emphasizing features of the ICA representations to distinguish the ICA representations in the depth images and binary shape images.

8. The human activity recognition method of claim 5 , wherein determining the codebook using vector quantization comprises:
classifying the features determined through the LDA into at least one group using the vector quantization; and
assigning a code to the at least one classified group.

9. The human activity recognition method of claim 5 , wherein determining observation symbol sequences in the video clip comprises:
determining codes of the video clip and a code arrangement order using codes of the codebook.

10. A human activity recognition apparatus comprising:
an input interface configured to receive a video clip;
a controller configured to determine principal components of depth images and binary shape images of human activities extracted from the video clip, and for performing a human activity modeling based on the principal components, and determine a human activity model which maximizes a likelihood probability among activity models in a dictionary according to an output value of the human activity modeling; and
an output interface configured to output the human activity model,
wherein the depth images of the human activities are different images than the binary shape images of the human activities, and the video clip comprises successive images that are based on a video sensor.

11. The human activity recognition apparatus of claim 10 , wherein the controller is configured to determine the principal components of the depth images and binary shape images of the human activities extracted from the video clip by determining at least one principal component representing an entire image based on a human body representation of the depth images and binary shape images.

12. The human activity recognition apparatus of claim 10 , wherein the video clip is represented using a HMM based on the status transition and a symbol observation matrix.

13. The human activity recognition apparatus of claim 10 , wherein the controller is configured to determine the human activity model which maximizes the likelihood probability from a human activity HMM dictionary.

14. The human activity recognition apparatus of claim 10 , wherein the controller is configured to determine ICA representations based on the principal components, determine features through LDA based on the ICA representations, determine a codebook using vector quantization, determine observation symbol sequences in the video clip, and identify human activities using a HMM based on a status transition and an observation matrix.

Assignees

• Samsung Electronics Co Ltd
• Univ-Industry Coop Group Of Kyung Hee Univ

Inventors

• Kim Hyun-Soo
• Lee Jong-Chang
• Sim Dae-Hyun
• Kim Tae-Seong

Classifications

• G06V10/85

  Markov-related models; Markov random fields · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title

• G06F18/295

  Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models · CPC title

• G06V40/176 Primary

  Dynamic expression · CPC title