Detection, tracking, and pose estimation of an articulated body

Patent metadata

Field	Value
Publication number	US-9911219-B2
Application number	US-201514749303-A
Country	US
Kind code	B2
Filing date	Jun 24, 2015
Priority date	May 13, 2015
Publication date	Mar 6, 2018
Grant date	Mar 6, 2018

Abstract

Techniques related to pose estimation for an articulated body are discussed. Such techniques may include extracting, segmenting, classifying, and labeling blobs, generating initial kinematic parameters that provide spatial relationships of elements of a kinematic model representing an articulated body, and refining the kinematic parameters to provide a pose estimation for the articulated body.

First claim

What is claimed is:

1. A method for generating a pose estimation for an articulated body comprising:
classifying a segmented blob as a hand blob;
generating one or more sets of finger labels for the hand blob;
generating, based at least in part on the labeled hand blob, a plurality of sets of initial kinematic model parameters that each provides spatial relationships of elements of a kinematic model representing an articulated body;
evaluating each of the plurality of sets of initial kinematic model parameters to determine a predetermined number of selected sets of initial kinematic model parameters, wherein the predetermined number of selected sets is less than the number evaluated sets of initial kinematic model parameters;
applying a kinematic model refinement to each of the selected sets of initial kinematic model parameters based on matching the kinematic model to target positions of the hand blob to generate multiple sets of resultant kinematic model parameters;
selecting a final resultant set of the multiple sets of resultant kinematic model parameters based on a scoring of the multiple sets of resultant kinematic model parameters; and
outputting the selected resultant set of kinematic model parameters.

2. The method of claim 1 , further comprising:
extracting the segmented blob from input depth image data, wherein extracting the segmented blob comprises:
an over-segmentation of the input depth image data to generate super-pixels,

wherein the super-pixels comprise regions of the input depth image data having depth gradients less than a predefined threshold; and
merging or splitting the super-pixels to generate the segmented blob.

3. The method of claim 1 , wherein classifying the segmented blob comprises applying a random forest classifier to a plurality of invariant features associated with the segmented blob.

4. The method of claim 1 , further comprising:
matching the hand blob to a previous frame hand blob, wherein the hand blob is associated with a current frame and the previous frame hand blob is associated with a previous frame, and wherein generating the one or more sets of finger labels for the hand blob comprises copying finger labels from a previous hand model associated with the previous frame hand blob.

5. The method of claim 1 , further comprising:
applying a rigid transformation to a second kinematic hand model, wherein the hand blob is associated with a current frame and the second kinematic hand model is associated with a previous frame, and wherein the second kinematic hand model comprises second finger labels; and
generating, based on the transformed second kinematic hand model, a second set of initial kinematic model parameters, wherein the plurality of sets of initial kinematic model parameters for evaluation comprises the second set of initial kinematic model parameters.

6. The method of claim 1 , wherein a first set of initial kinematic model parameters corresponds to a first set of the one or more sets of finger labels and a second set of initial kinematic model parameters corresponds to a second set of the one or more sets of finger labels, wherein the first set of finger labels provides a first finger label to a first portion of the hand blob and the second set of finger labels provides a second finger label different than the first finger label to the first portion of the hand blob.

7. The method of claim 1 , wherein the scoring comprises a comparison of multiple errors, each associated with one of the multiple sets of resultant kinematic model parameters, wherein each of the multiple errors is based on comparing the kinematic model implementing a corresponding one of the multiple sets of resultant kinematic model parameters with the hand blob.

8. The method of claim 1 , wherein generating at least one of the one or more sets the finger labels comprises:
applying edge detection to the hand blob;
locating bases and tips of a plurality of portions of the hand blob;
generating invariant feature vectors associated with the portions of the hand blob; and
classifying the hand blob as representing a left hand or a right hand based on the one or more invariant feature vectors.

9. The method of claim 8 , further comprising:
providing one or more finger labels to each of the portions of the hand blob; and
generating, based on the one or more finger labels, at least some of the plurality of sets of initial kinematic model parameters based on a plurality of finger label permutations associated with the hand blob.

10. The method of claim 1 , wherein generating the finger labels comprises:
applying edge detection to the hand blob;
locating a base and a tip of a first portion of the hand blob;
generating an invariant feature vector associated with the first portion of the hand blob, the invariant feature vector comprising a width of the first portion, a length of the first portion and, for each of a plurality of cells of an image grid centered at the base of the first portion, a major edge orientation and a depth difference between an average depth of the cell and a reference depth; and
providing a finger label to the first portion of the hand blob based on applying a classifier to the invariant feature vector.

11. The method of claim 1 , wherein the kinematic model refinement comprises at least one of a particle swarm optimization technique, a Levenberg Marquardt technique based on a numerical Jacobian, a partial Levenberg Marquardt technique, or an inverse kinematics based iterative closest point technique.

12. The method of claim 1 , wherein the kinematic model comprises a pose based on initial kinematic model parameters that provide spatial relationships of elements of the kinematic model, and wherein the kinematic model comprises a plurality of joints, a plurality of end-effectors, and links between selected joints and end-effectors all within a model skin of the kinematic model, and wherein applying the kinematic model refinement to a first set of initial kinetic model parameters comprises:
selecting, based on the hand blob, target positions for matching the kinematic model;
generating, in addition to the end-effectors of the kinematic model, a plurality of virtual end-effectors corresponding to the target positions based on the plurality of target positions and the kinematic model, wherein each of the virtual end-effectors is generated at a point on the model skin of the kinematic model closest to an associated target position of the target positions;
generating an inverse kinematics problem comprising a Jacobian matrix based on the first set of initial kinematic model parameters, the target positions, and the virtual end-effectors;
determining a change in the first set of kinematic model parameters based on the inverse kinematics problem; and
repeating the selecting the plurality of target positions, generating the plurality of virtual end-effectors, generating the inverse kinematics problem, and determining the change in the first set of kinematic model parameters until a convergence is attained to generate a first set of resultant kinematic model parameters.

13. The method of claim 12 , wherein the Jacobian matrix comprises at least one of an element having a target weighting parameter associated with a first target position of the plurality of target positions, an element having a joint weighting parameter associated with a first joint of the elements of the kinematic model, or an element having a repulsive target functionality associated with a first target position of the plurality of target positions.

14. The method of claim 12 , wherein the inverse kinematics model co

Assignees

• Intel Corp

Inventors

• Fleishman Shahar
• Kliger Mark
• Lerner Alon

Classifications

• G06T7/75

  involving models · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T7/143

  involving probabilistic approaches, e.g. Markov random field [MRF] modelling · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title