Method and apparatus for expressing motion object

Patent metadata

Field	Value
Publication number	US-9589365-B2
Application number	US-201514629650-A
Country	US
Kind code	B2
Filing date	Feb 24, 2015
Priority date	Feb 27, 2014
Publication date	Mar 7, 2017
Grant date	Mar 7, 2017

Abstract

A method and an apparatus for expressing a motion object are disclosed. The method includes obtaining a stereo image in which the motion object has been captured, the stereo image including a depth image; extracting a key point from the motion object in the stereo image; determining, based on statistical information relating to three-dimensional motion of pixels within a first predetermined region surrounding the key point, a dominant direction of the key point; determining, based on the dominant direction of the key point, motion vectors of pixels within a second predetermined region surrounding the key point to obtain rotation invariant motion vectors; and extracting, based on the determined motion vectors of the pixels within the second predetermined region surrounding the key point, a feature describing the key point. The present invention can extract features of motion object that are irrelevant to a viewing angle of a camera.

First claim

What is claimed is:

1. A method for expressing a motion object, the method comprising:
obtaining a stereo image of the motion object, the stereo image including a depth image;
extracting a key point from the motion object in the stereo image;
determining a dominant direction of the key point, based on statistical information associated with three-dimensional motion of pixels within a first region surrounding the key point wherein determining the dominant direction includes,
obtaining a two-dimensional histogram having a first dimension and a second dimension, the first dimension being a depth of motion vectors of the pixels and the second dimension being directions of components of the motion vectors of the pixels in planes perpendicular to the depth, and
determining the dominant direction of the key point based on a direction relating to a peak value in the two-dimensional histogram;

determining, based on the dominant direction of the key point, motion vectors of pixels within a second region surrounding the key point to obtain rotation invariant motion vectors; and
generating, based on the determined motion vectors of the pixels within the second region surrounding the key point, a feature associated with the key point; and
generating a classification signal based on the generated feature, the classification signal including information indicating a classification of the motion object.

2. The method for expressing a motion object according to claim 1 , wherein extracting the key point from the motion object in the stereo image includes
obtaining, from the depth image, a motion region representing a motion part of the motion object, and
extracting the key point from the motion region.

3. The method for expressing a motion object according to claim 2 , wherein obtaining the motion region representing the motion part of the motion object from the depth image includes
performing a division of the depth image and previous temporal depth images to obtain a sequence of motion object regions,
taking statistics of appearance frequencies of pixels at each position in the motion object regions, and
determining and removing, based on the appearance frequencies of the pixels, substantially stationary parts from the motion object regions.

4. The method for expressing a motion object according to claim 1 , wherein the generating the feature associated with the key point based on the determined motion vectors of the pixels within the second region surrounding the key point includes
calculating a motion direction histogram of the determined motion vectors of the pixels in the second region, and
generating the feature associated with the key point based on the motion direction histogram.

5. The method for expressing a motion object according to claim 1 , the method further comprising:
for one or more of temporal sample depth images of the motion object in at least one motion period, performing operations of extracting the key point, determining the dominant direction of the key point, rotating the motion vectors of the pixels and generating the feature associated with the key point, and obtaining a set of features of the key point for training; and
training a classifier based on the set of features of the key point for training.

6. The method for expressing a motion object according to claim 5 , the method further comprising:
receiving one or more of temporal test depth images of the motion object in at least one motion period, performing operations of extracting the key point, determining the dominant direction of the key point, rotating the motion vectors of the pixels and generating the feature associated with the key point, and obtaining a set of the features of the key point for testing; and
classifying, based on the set of features of the key point for testing and the trained classifier, the motion object relating to the test depth images.

7. The method for expressing a motion object according to claim 1 , wherein determining the motion vectors of the pixels within the second region surrounding the key point based on the dominant direction of the key point includes
rotating an image reference coordinate system based on the dominant direction of the key point, and
determining the second region surrounding the key point in the rotated image reference coordinate system.

8. The method for expressing a motion object according to claim 7 , wherein determining the motion vectors of the pixels within the second region surrounding the key point includes
for the pixels whose motion vectors have been calculated, converting the calculated motion vectors and obtaining the motion vectors of the pixels in the rotated image reference coordinate system, and
for the pixels whose motion vectors have not been calculated, calculating the motion vectors of the pixels in the rotated image reference coordinate system.

9. An apparatus for expressing a motion object, the apparatus comprising:
a memory configured to store computer-readable instructions; and
a processor configured to execute the computer-readable instructions to,
obtain a stereo image in which the motion object has been captured, the stereo image including a depth image;
extract a key point from the motion object in the stereo image;
determine, based on statistical information relating to three-dimensional motion of pixels within a first region surrounding the key point, a dominant direction of the key point, the determining the dominant direction including,
obtaining a two-dimensional histogram having a first dimension and a second dimension, the first dimension being a depth of motion vectors of the pixels and the second dimension being directions of components of the motion vectors of the pixels in planes perpendicular to the depth, and
determining the dominant direction of the key point based on a direction relating to a peak value in the two-dimensional histogram;

determine, based on the dominant direction of the key point, motion vectors of pixels within a second region surrounding the key point; and
generate, based on the determined motion vectors of the pixels within the second region surrounding the key point, a feature associated with the key point; and
generate a classification signal based on the generated feature, the classification signal including information indicating a classification of the motion object.

10. The apparatus for expressing a motion object according to claim 9 , wherein the processor is configured to execute the computer-readable instructions to,
obtain, from the depth image, a motion region representing a motion part of the motion object, and
extract the key point from the motion region.

11. The apparatus for expressing a motion object according to claim 10 , wherein the processor is configured to execute the computer-readable instructions to
perform a division of the depth image and previous temporal depth images to obtain a sequence of motion object regions,
take statistics of appearance frequencies of pixels at each position in the motion object regions, and
determine and remove, based on the appearance frequencies of the pixels, substantially stationary parts from the motion object regions.

12. The apparatus for expressing a motion object according to claim 9 , wherein the processor is configured to execute the computer-readable instructions to
calculate a motion direction histogram of the determined motion vectors of the pixels in the second region, and
generate the feature associated with the key point based on the motion direction histogram.

13. The apparatus for expressing a motion object according to claim 9 , wherein the pr

Assignees

• Zhao Ying
• Ricoh Co Ltd

Inventors

• Zhao Ying

Classifications

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06K9/00348

  Physics · mapped topic

• G06T7/2053

  Physics · mapped topic