Systems and methods of tracking moving hands and recognizing gestural interactions

Patent metadata

Field	Value
Publication number	US-9785247-B1
Application number	US-201514712699-A
Country	US
Kind code	B1
Filing date	May 14, 2015
Priority date	May 14, 2014
Publication date	Oct 10, 2017
Grant date	Oct 10, 2017

Abstract

The technology disclosed relates to relates to providing command input to a machine under control. It further relates to gesturally interacting with the machine. The technology disclosed also relates to providing monitoring information about a process under control. The technology disclosed further relates to providing biometric information about an individual. The technology disclosed yet further relates to providing abstract features information (pose, grab strength, pinch strength, confidence, and so forth) about an individual.

First claim

What is claimed is:

1. A method of determining command input to a machine responsive to control object gestures in three dimensional (3D) sensory space, the method comprising:
determining observation information including gestural motion of a control object in three dimensional (3D) sensory space from at least one image captured at time t0;
constructing a 3D model to represent the control object by fitting one or more 3D capsules to the observation information based on the image captured at time t0;
responsive to modifications in the observation information based on another image captured at time t1, wherein the control object moved between t0 and t1, improving alignment of the 3D capsules to the modified observation information by:
determining variance between a point on another set of observation information based on the image captured at time t1 and a corresponding point on at least one of the 3D capsules fitted to the observation information based on the image captured at time t0 by:
pairing point sets on an observation information of the control object with points on axes of the 3D capsules, wherein the observation information points lie on vectors that are normal to the axes; and
determining a reduced root mean squared deviation (RMSD) of distances between paired point sets; and

responsive to the variance adjusting the 3D capsules and determining a gesture performed by the control object based on the adjusted 3D capsules; and
interpreting the gesture as providing command input to a machine under control.

2. The method of claim 1 , wherein adjusting the 3D capsules further includes improving conformance of the 3D capsules to at least one of length, width, orientation, and arrangement of portions of the observation information.

3. The method of claim 1 , further including:
receiving an image of a hand as the control object;
determining span modes of the hand, wherein the span modes include at least a finger width span mode and a palm width span mode; and
using span width parameters for the finger width and palm width span modes to initialize 3D capsules of a 3D model of the hand.

4. The method of claim 1 , further including:
receiving an image of a hand as the control object;
determining span modes of the hand, wherein the span modes include at least a finger width span mode, a palm width span mode, and a wrist width span mode; and
using span width parameters for the finger width, palm width, and wrist width span modes to initialize a 3D model of the hand and corresponding arm.

5. The method of claim 1 , further including interpreting the gesture as selecting one or more heterogeneous devices in the 3D sensory space.

6. The method of claim 1 , further including interpreting the gesture as selecting one or more heterogeneous marker images that trigger augmented illusions.

7. The method of claim 1 , further including automatically switching the machine under control from one operational mode to another in response to interpreting the gesture.

8. The method of claim 1 , wherein determining the variance further includes determining whether the point on another set of observation information based on the image captured at time t1 and the corresponding point on one of the 3D capsules fitted to the observation information defined based on the image captured at time t0 are within a threshold closest distance.

9. The method of claim 1 , wherein determining the variance further includes:
pairing point sets on an observation information of the control object with points on the 3D capsules, wherein normal vectors to the points sets are parallel to each other; and
determining a reduced root mean squared deviation (RMSD) of distances between bases of the normal vectors.

10. The method of claim 1 , further including determining from the 3D model at least one of a velocity of a portion of a hand, a state, a pose.

11. The method of claim 10 , wherein the determining a velocity further includes determining at least one of a velocity of one or more fingers, and a relative motion of a portion of the hand.

12. The method of claim 10 , wherein the determining a state further includes determining at least one of a position, an orientation, and a location of a portion of the hand.

13. The method of claim 10 , wherein the determining a pose further includes determining at least one of whether one or more fingers are extended or non-extended, one or more angles of bend for one or more fingers, a direction to which one or more fingers point, a configuration indicating a pinch, a grab, an outside pinch, and a pointing finger.

14. The method of claim 10 , further including determining from the 3D model whether a tool or object is present in the hand.

15. The method of claim 1 , further comprising:
determining gesture features of the control object based on the 3D capsules; and
issuing a feature-specific command input to a machine under control based on the determined gesture features.

16. The method of claim 15 , wherein the control object is a hand and the gesture features include edge information for fingers of the hand.

17. The method of claim 15 , wherein the control object is a hand and the gesture features include edge information for palm of the hand.

18. The method of claim 15 , wherein the control object is a hand and the gesture features include joint angle and segment orientation information of the hand.

19. The method of claim 15 , wherein the control object is a hand and the gesture features include finger segment length information for fingers of the hand.

20. The method of claim 15 , wherein the control object is a hand and the gesture features include curling of the hand during the gestural motion.

21. The method of claim 15 , wherein the control object is a hand and the gesture features include at least one of a pose, a grab strength, a pinch strength and a confidence of the hand.

22. The method of claim 1 , further comprising:
determining biometric features of the control object based on the 3D capsules;
authenticating the control object based on the determined biometric features;
upon determining the command input indicated by the gestural motion of the control object, determining whether the authenticated control object is authorized to issue the command input; and
issuing an authorized command input to a machine under control.

23. The method of claim 22 , wherein the control object is a hand and the determined biometric features include at least one of measurements across a palm of the hand and finger width at a first knuckle of the hand.

24. A non-transitory computer readable storage medium impressed with computer program instructions to determine command input to a machine responsive to control object gestures in three dimensional (3D) sensory space, the instructions, when executed on a processor, implement actions comprising:
determining observation information including gestural motion of a control object in three dimensional (3D) sensory space from at least one image captured at time t0;
constructing a 3D model to represent the control object by fitting one or more 3D capsules to the observation information based on the image captured at time t0;
responsive to modifications in the observation information based on another image captured at time t1, wherein the control object moved between t0 and t1, improving alignment of the 3D capsules to the modified observation information by:
determining variance bet

Assignees

• Leap Motion Inc

Inventors

• Horowitz Kevin A
• Perez Matias
• Bedikian Raffi
• Holz David S
• Hare Gabriel A

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title