Learning based estimation of hand and finger pose

Patent metadata

Field	Value
Publication number	US-9330307-B2
Application number	US-201514636281-A
Country	US
Kind code	B2
Filing date	Mar 3, 2015
Priority date	Sep 11, 2011
Publication date	May 3, 2016
Grant date	May 3, 2016

Abstract

A method for processing data includes receiving a depth map of a scene containing a human hand, the depth map consisting of a matrix of pixels having respective pixel depth values. The method continues by extracting from the depth map information based on the depth values in a plurality of positions distributed over the human hand and processing the information in order to estimate respective candidate positions of the finger joints. The pose of the human hand is estimated by choosing a combination of the positions of the finger joints, responsively to anatomical constraints of the hand, that gives a hand configuration that is most anatomically probable among the candidate positions.

First claim

The invention claimed is:

1. A method for processing data, comprising:
receiving a depth map of a scene containing a human hand, the depth map comprising a matrix of pixels having respective pixel depth values, the hand including fingers having multiple joints;
extracting from the depth map information based on the depth values in a plurality of positions distributed over the human hand;
processing the information in order to estimate respective candidate positions of the finger joints; and
estimating a pose of the human hand by choosing a combination of the positions of the finger joints, responsively to anatomical constraints on bending angles of the finger joints, that gives a hand configuration that is most anatomically probable among the candidate positions.

2. The method according to claim 1 , wherein estimating the pose comprises applying kinematics based on the anatomical constraints in processing the information.

3. The method according to claim 1 , wherein estimating the pose comprises expressing the pose in terms of a hand posture description language.

4. The method according to claim 3 , wherein the hand posture description language comprises assigning one of a set of positions to each finger of the hand so as to define postures of the hand.

5. The method according to claim 4 , wherein defining postures of the hand comprises defining one or more invalid postures of the hand, and wherein estimating the pose of the hand comprises excluding the one or more invalid postures.

6. The method according to claim 1 , and comprising normalizing a depth of the depth map by finding a representative depth coordinate of the human hand in the depth map and projecting a point cloud derived from the depth map responsively to the representative depth coordinate, and applying the normalized depth in estimating the pose.

7. The method according to claim 1 , wherein estimating the pose comprises finding respective locations of landmarks of the human hand, and wherein the method comprises calibrating a scale of the human hand by finding a distance between the locations of the landmarks and scaling the depth map responsively to the distance, and applying the calibrated scale in estimating the pose.

8. The method according to claim 1 , wherein receiving the depth map comprises receiving a sequence of depth maps, and wherein estimating the pose comprises tracking movement of the human hand over multiple frames in the sequence.

9. The method according to claim 8 , and comprising controlling a computer application responsively to the tracked movement.

10. Mapping apparatus, comprising:
an imaging assembly, which is configured to provide a depth map of a scene containing a human hand, the depth map comprising a matrix of pixels having respective pixel depth values, the hand including fingers having multiple joints; and
a processor, which is configured to extract from the depth map information based on the depth values in a plurality of positions distributed over the human hand, to process the information in order to estimate respective candidate positions of the finger joints, and to estimate a pose of the human hand by choosing a combination of the positions of the finger joints, responsively to anatomical constraints on bending angles of the finger joints, that gives a hand configuration that is most anatomically probable among the candidate positions.

11. The apparatus according to claim 10 , wherein the processor is configured to apply kinematics based on the anatomical constraints in processing the descriptors.

12. The apparatus according to claim 10 , wherein the processor is configured to express the pose in terms of a hand posture description language.

13. The apparatus according to claim 12 , wherein the hand posture description language assigns one of a set of positions to each finger of the hand so as to define postures of the hand.

14. The apparatus according to claim 13 , wherein the processor is configured to define one or more invalid postures of the hand, and to exclude the one or more invalid postures in estimating the pose of the hand.

15. The apparatus according to claim 10 , wherein the processor is configured to normalize a depth of the depth map by finding a representative depth coordinate of the human hand in the depth map and to project a point cloud derived from the depth map responsively to the representative depth coordinate, and to apply the normalized depth in estimating the pose.

16. The apparatus according to claim 10 , wherein the processor is configured to find respective locations of landmarks of the human hand, to calibrate a scale of the human hand by finding a distance between the locations of the landmarks, to scale the depth map responsively to the distance, and to apply the calibrated scale in matching the descriptors and in estimating the pose.

17. The apparatus according to claim 10 , wherein the imaging assembly is configured to provide a sequence of depth maps, and wherein the processor is configured to track movement of the human hand over multiple frames in the sequence.

18. The apparatus according to claim 17 , wherein the processor is configured to control a computer application responsively to the tracked movement.

19. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a depth map of a scene containing a human hand, the depth map comprising a matrix of pixels having respective pixel depth values, the hand including fingers having multiple joints,
wherein the instructions cause the computer to extract from the depth map information based on the depth values in a plurality of positions distributed over the human hand, to process the information in order to estimate respective candidate positions of the finger joints, and to estimate a pose of the human hand by choosing a combination of the positions of the finger joints, responsively to anatomical constraints on bending angles of the finger joints, that gives a hand configuration that is most anatomically probable among the candidate positions.

20. The product according to claim 19 , wherein the instructions cause the computer to express the pose in terms of a hand posture description language.

Assignees

• Apple Inc

Inventors

• Litvak Shai
• Brailovsky Leonid
• Yanir Tomer

Classifications

• G06V40/11

  Hand-related biometrics; Hand pose recognition · CPC title

• G06V20/64 Primary

  Three-dimensional [3D] objects · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06K9/00382 Primary

  Physics · mapped topic

• G06K9/4671

  Physics · mapped topic