Activity recognition method using videotubes

Patent metadata

Field	Value
Publication number	US-10628667-B2
Application number	US-201815867932-A
Country	US
Kind code	B2
Filing date	Jan 11, 2018
Priority date	Jan 11, 2018
Publication date	Apr 21, 2020
Grant date	Apr 21, 2020

Abstract

An activity recognition device comprises a port configured to receive a video stream from a video source for a first object and a second object; a memory configured to store instructions and image frames of the video stream; and one or more processors, wherein the one or more processors execute the instructions stored in the memory, the one or more processors configured to: select portions of the image frames based on presence of the first object; determine areas within the portions of image frames, wherein locations of the first object in the video frames are bounded by the determined areas; determine motion of the first object and locations of a second object within the areas of the image frames; and identify an activity according to the determined motion and locations of the second object, and generate an alert according to the identified activity.

First claim

What is claimed is:

1. An activity recognition device, the device comprising:
a port configured to receive a video stream from a video source for a first object and a second object;
a memory configured to store instructions and image frames of the video stream; and
one or more processors, wherein the one or more processors execute the instructions stored in the memory, the one or more processors configured to:
select portions of the image frames based on presence of the first object;
determine minimized areas within the portions of image frames that include information of motion of the first object and location of a second object;
determine motion of the first object and locations of the second object within the minimized areas of a series of the image frames, wherein the order of the series of the multiple frames includes temporal information of the first object and the second object; and
identify an activity according to the determined motion and locations of the second object using minimized area of the series of image frames, and generate an alert according to the identified activity.

2. The activity recognition device of claim 1 , wherein the one or more processors are configured to:
determine a similarity score between a first windowed portion of a first image frame and the same first windowed portion of a second image frame, wherein the active area is included in the first windowed portion of the first and second image frames;
omit processing of the first windowed portion of the second image frame when the similarity score is greater than a specified similarity threshold; and
when the similarity score is less than the specified similarity threshold, perform detection of the first object in the second image frame to generate a second windowed portion of the image frames of the video stream more likely to include an image of the first object than other portions of the image frames, and include the second windowed portion of the image in a videotube that includes a collection of rearranged portions of the image frames of the video stream.

3. The activity recognition device of claim 1 , wherein the one or more processors are configured to recurrently set a window size of the active area, wherein the window size is set to include the first object.

4. The activity recognition device of claim 3 , wherein the first object is a hand and the wherein the one or more processors are configured to:
determine a center of a hand active area;
identify a search area by scaling a boundary of the hand active area with respect to the determined center;
perform hand image detection in the identified search area; and
set the size window according to a result of the hand image detection.

5. The activity recognition device of claim 3 , wherein the one or more processors are configured to:
use the determined motion of the first object to predict a next window;
perform image detection of the first object using the next window;
replace a current window with the next window when the next window contains the boundaries of a detected image of the first object; and
when the boundaries of the detected image of the first object extends beyond the next window:
merge the current window and the next window;
identify an image of the first object in the merged windows; and
determine a new minimized window size that contains the identified image of the first object.

6. The activity recognition device of claim 1 , wherein the first object is a hand, and the wherein the one or more processors are configured to:
identify pixels of the determined areas that include a hand image; and
track a change in pixels that include the hand image between windowed portions of the image frames to determine hand motion.

7. The activity recognition device of claim 1 , wherein the first object is a hand, and wherein the one or more processors are configured to:
determine locations of fingertips and joint points in the image frames; and
track the change in fingertips and joint points between windowed portions of the image frames to determine the hand motion.

8. The activity recognition device of claim 1 , wherein the first object is a hand, and wherein the one or more processors are configured to:
determine a hand motion; and
identify the activity using the determined hand motion and the second object.

9. The activity recognition device of claim 8 , wherein the one or more processors are further configured to:
compare a combination of the determined hand motion and second object to one or more combinations of hand motions and objects stored in the memory; and
identify the activity based on a result of the comparison.

10. The activity recognition device of claim 8 , wherein the one or more processors are further configured to:
detect a sequence of hand motions using the determined areas of the image frames;
compare the detected sequence of hand motions to a specified sequence of hand motions of one or more specified activities; and
select an activity from the one or more specified activities according to a result of the comparing.

11. The activity recognition device of claim 1 , wherein the one or more processors are further configured to generate a videotube that includes a collection of rearranged portions of the image frames of the video stream that include the first and second object, and corresponding feature maps.

12. The activity recognition device of claim 11 , wherein the one or more processors are configured to store videotube information in the memory as a scalable tensor videotube; and wherein the activity classifier component is configured to apply the scalable tensor videotube as input to a deep learning algorithm performed by the activity classifier component to identify the activity of the person.

13. The activity recognition device of claim 12 , wherein the one or more processors are configured to select a row-wise configuration of portions of the image frames within the scalable tensor videotube according to the identity of the person and apply the selected row-wise configuration as the input to the deep learning algorithm to identify the activity of the person.

14. The activity recognition device of claim 12 , wherein the one or more processors are configured to select a column-wise configuration of portions of the image frames within the scalable tensor videotube according to identities of multiple persons and apply the selected column-wise configuration as the input to the deep learning algorithm to identify an interactivity between the multiple persons.

15. The activity recognition device of claim 12 , wherein the one or more processors are configured to select multiple column-wise configuration of portions of the image frames within the scalable tensor videotube according to identities of multiple groups of persons and apply the selected multiple column-wise configuration as the input to the deep learning algorithm to identify multiple interactivities between the multiple groups of persons.

16. The activity recognition device of claim 1 , wherein the video source includes an imaging array configured to provide a video stream of an image of a vehicle compartment; and wherein the one or more processors are included in a vehicle processing unit configured to identify an activity using the video stream of the image of the vehicle compartment.

17. A computer-implemented method of machine recognition of an activity, the method comprising:
obtaining a video stream of a first object and a second object using a video source;
selecting portions of image frames of the video

Assignees

• Futurewei Technologies Inc

Inventors

• Porikli Fatih
• Xu Qijie
• Bill Luis
• Wei Huang

Classifications

• G06T7/11

  Region-based segmentation · CPC title

• B60Y2302/03

  Actuating a signal or alarm device · CPC title

• G06K9/00845

  Physics · mapped topic

• G06K9/3233

  Physics · mapped topic

• G06K9/00389 Primary

  Physics · mapped topic