Non-linear motion capture using Frenet-Serret frames

Patent metadata

Field	Value
Publication number	US-9857876-B2
Application number	US-201414338136-A
Country	US
Kind code	B2
Filing date	Jul 22, 2014
Priority date	Jul 22, 2013
Publication date	Jan 2, 2018
Grant date	Jan 2, 2018

Abstract

Implementations of the technology disclosed convert captured motion from Cartesian/(x,y,z) space to Frenet-Serret frame space, apply one or more filters to the motion in Frenet-Serret space, and output data (for display or control) in a desired coordinate space—e.g., in a Cartesian/(x,y,z) reference frame. The output data can better represent a user's actual motion or intended motion.

First claim

What is claimed is:

1. A method of accurately capturing a non-linear gestural path of a hand-gesture in three-dimensional (3D) sensory space, the method including:
capturing with a camera a sequence of multiple images of a user's hand making a non-linear free-form hand-gesture performed by the user's hand, moving freely and substantially independently of contact with other objects in a three-dimensional (3D) sensory space monitored by the camera;
using a first coordinate system associated with the camera, determining from the sequence of multiple images a first set of coordinates of a plurality of 3D positions along a hand-gesture movement path made by the user's hand during the non-linear free-form hand-gesture captured;
determining an orientation invariant trajectory for the hand-gesture movement path during the non-linear free-form hand-gesture by transforming the first set of coordinates of the plurality of 3D positions along the hand-gesture movement path captured in the first coordinate system into a second set of coordinates in a second coordinate system according to a transform defining a relationship between the first coordinate system and the second coordinate system, including:
transforming points in the images captured representing positions in space of portions of the user's hand captured using the first coordinate system corresponding to coordinates of the camera into corresponding points using the second coordinate system by attaching to select ones of the points a frame of reference consisting of an orthonormal vector set and
building a sequence of orthonormal vector sets at sequential points of the select ones of the points thereby describing motion of the user's hand at the select ones of the points as the sequence of orthonormal vector sets;

applying at least one filter to the second set of coordinates to obtain filtered curvilinear motion data for the hand-gesture movement path;
comparing the orientation invariant trajectory for the hand-gesture movement path to a library of known gestures electronically stored as records in a database to detect a gesture; and when a similar trajectory for a user's hand located in the library of known gesture is found, providing a command input to control a system based upon the gesture detected; and
generating for display a smoothened representation of the hand-gesture movement path using an approximate best-fit curve that predicts a trajectory for the user's hand by connecting points along the hand-gesture movement path.

2. The method of claim 1 , further including:
recognizing a second hand-gesture as corresponding to a recording of the hand-gesture movement path by:
determining an orientation invariant trajectory of a second hand-gesture movement path of the second hand-gesture; and
matching the orientation invariant trajectory of the second hand-gesture movement path to the orientation invariant trajectory of the hand-gesture movement path; thereby eliminating from consideration translation or rotation of the second hand-gesture relative to the recording of the hand-gesture movement path.

3. The method of claim 1 , further including using the command input to control and interact with computer systems by moving fingers in air.

4. The method of claim 1 , further including using the command input to perform free-floating handwriting recognition.

5. The method of claim 4 , the free-floating handwriting recognition further including applying the smoothed representation to disambiguate between similarly shaped trajectories of handwriting characters in the 3D sensory space.

6. The method of claim 1 , further including the hand holding a stylus.

7. The method of claim 1 , wherein the orthonormal vector set includes: a tangent unit vector that is tangent to the hand-gesture movement path and pointing in a direction of motion of the user's hand at a corresponding point, a normal unit vector that is derivative of the tangent unit vector with respect to an arclength of at least a portion of the hand-gesture movement path divided by a length of the at least a portion of the hand-gesture movement path, and a binormal unit vector comprising a cross product of the tangent unit vector and the normal unit vector.

8. The method of claim 1 , further including associating a frame of reference with at least two locations along the hand-gesture movement path and having no corresponding point in the images captured.

9. The method of claim 8 , further including distributing the at least two locations evenly along the hand-gesture movement path.

10. The method of claim 8 , further including distributing additional frames of reference at locations along the hand-gesture movement path in which the hand-gesture movement path changes direction more rapidly than at any other points in the hand-gesture movement path.

11. The method of claim 1 , further including:
determining a curvature and a torsion of the hand-gesture movement path from a rotation between two consecutive ones of the sequence of orthonormal vector sets for the select ones of the points along the hand-gesture movement path; and
removing from the hand-gesture movement path, points indicated by the curvature and torsion to correspond to unintended motion; thereby creating a resultant hand-gesture movement path.

12. The method of claim 11 , further including:
replacing points removed from the hand-gesture movement path with a curve defined by Euler's spiral fitted to the hand-gesture movement path in the second coordinate system; and
transforming the hand-gesture movement path back to the first coordinate system.

13. The method of claim 12 , further including:
fitting a spiral having a curvature near zero at a tangent portion and increasing linearly with its curve length such that when the spiral meets a circular portion of the hand-gesture movement path, curvature of the spiral is equal to curvature of the hand-gesture movement path.

14. The method of claim 11 , further including:
using the resultant hand-gesture movement path, predicting an intended free-form hand-gesture for a user before the non-linear free-form hand-gesture has been completed.

15. The method of claim 1 , further including constructing for display an Euler spiral to model boundary segments between at least a first extreme and a second extreme of the hand-gesture movement path.

16. A non-transitory computer readable medium storing instructions to accurately capture a non-linear gestural path of a hand-gesture in three-dimensional (3D) sensory space, which instructions when executed by a processor perform actions including:
capturing with a camera a sequence of multiple images of a user's hand making a non-linear free-form hand-gesture performed by the user's hand, moving freely and substantially independently of contact with other objects in a three-dimensional (3D) sensory space monitored by the camera;
using a first coordinate system associated with the camera, determining from the sequence of multiple images a first set of coordinates of a plurality of 3D positions along a hand-gesture movement path made by the user's hand during the non-linear free-form hand-gesture captured;
determining an orientation invariant trajectory for the hand-gesture movement path during the non-linear free-form hand-gesture by transforming the first set of coordinates of the plurality of 3D positions along the hand-gesture movement path captured in the first coordinate system into a second set of coordinates in a second coordinate system according to a transform defining a relationship between the first coordinate system and the second co

Assignees

• Leap Motion Inc

Inventors

• Hare Gabriel A
• Mertens Keith
• Perez Matias
• Roy Neeloy
• Holz David

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title