Dynamic, free-space user interactions for machine control

Patent metadata

Field	Value
Publication number	US-9459697-B2
Application number	US-201414155722-A
Country	US
Kind code	B2
Filing date	Jan 15, 2014
Priority date	Jan 15, 2013
Publication date	Oct 4, 2016
Grant date	Oct 4, 2016

Abstract

Embodiments of display control based on dynamic user interactions generally include capturing a plurality of temporally sequential images of the user, or a body part or other control object manipulated by the user, and computationally analyzing the images to recognize a gesture performed by the user. In some embodiments, a scale indicative of an actual gesture distance traversed in performance of the gesture is identified, and a movement or action is displayed on the device based, at least in part, on a ratio between the identified scale and the scale of the displayed movement. In some embodiments, a degree of completion of the recognized gesture is determined, and the display contents are modified in accordance therewith. In some embodiments, a dominant gesture is computationally determined from among a plurality of user gestures, and an action displayed on the device is based on the dominant gesture.

First claim

What is claimed is:

1. A method of controlling a machine, comprising:
sensing a variation of position of at least one control object using an imaging system;
determining from the variation one or more primitives describing at least one of motion made in space by the control object and character of the control object;
comparing the one or more primitives to one or more templates in a library of gesture templates;
selecting from a result of the comparing a set of templates of possible gestures corresponding to the one or more primitives; and
providing at least one of the set of templates of possible gestures as an indication of a command to issue to a machine under control responsive to the variation.

2. A method according to claim 1 , wherein the sensing a variation of position of at least one control object using an imaging system comprises:
capturing a plurality of temporally sequential images of at least one control object manipulated by the user.

3. A method according to claim 2 , wherein the determining from the variation one or more primitives describing at least one of motion made by the control object and character of the control object comprises:
computationally analyzing the images of the at least one control object to recognize a gesture primitive including at least a portion of a trajectory (trajectory portion) describing motion made by the control object.

4. A method according to claim 3 , wherein the computationally analyzing the images of the at least one control object to recognize a gesture primitive comprises:
identifying a scale associated therewith, the scale being indicative of an actual distance traversed by the control object; and
computationally determining a ratio between the scale and a displayed movement corresponding to an action to be displayed on a presentation device;
displaying the action on the device based on the ratio; and
adjusting the ratio based on an external parameter.

5. The method according to claim 4 , wherein the scale is identified by comparing the recognized gesture with records in a gesture database, the gesture database comprising a series of electronically stored records each relating a gesture to an input parameter.

6. The method according to claim 5 , wherein the gestures are stored in the records as vectors.

7. The method according to claim 4 , wherein the external parameter is the actual gesture distance.

8. The method according to claim 4 , wherein the external parameter is a ratio of a pixel distance in the captured images corresponding to performance of the gesture to a size, in pixels, of the display screen.

9. The method according to claim 3 , wherein analyzing the images of the at least one control object comprises (i) identifying a shape and position of the at least one control object in the images and (ii) reconstructing the position and the shape of the at least one control object in 3D space based on correlations between the identified shapes and positions of the at least one control object in the images.

10. The method according to claim 9 , further comprising defining a 3D model of the at least one control object and reconstructing the position and shape of the at least one control object in 3D space based on the 3D model.

11. The method according to claim 9 , wherein analyzing the images of the at least one control object further comprises temporally combining the reconstructed positions and shapes of the at least one control object in 3D space.

12. A method according to claim 1 , wherein comparing the one or more primitives to one or more templates in a library of gesture templates comprises:
disassembling at least a portion of a trajectory into a set of frequency components; and
searching for the set of frequency components among the template(s) stored in the library.

13. A method according to claim 12 , wherein disassembling at least a portion of a trajectory into a set of frequency components comprises:
applying Fourier analysis to the trajectory portion as a signal over time to determine the set of frequency components.

14. A method according to claim 1 , wherein comparing the one or more primitives to one or more templates in a library of gesture templates comprises:
disassembling at least a portion of a trajectory into a set of frequency components;
fitting a set of one or more functions to a set of frequency components representing at least a portion of a trajectory; and
searching for the set of functions among the template(s) stored in the library.

15. A method according to claim 14 , wherein fitting a set of one or more functions to a set of frequency components representing at least a portion of a trajectory comprises:
fitting Gaussian function to the set of frequency components.

16. A method according to claim 1 , wherein comparing the one or more primitives to one or more templates in a library of gesture templates comprises:
disassembling at least a portion of a trajectory into a set of time dependent frequency components; and
searching for the set of time dependent frequency components among the template(s) stored in the library.

17. A method according to claim 16 , wherein disassembling at least a portion of a trajectory into a set of time dependent frequency components comprises:
applying wavelet analysis to the trajectory portion as a signal over time to determine the set of time dependent frequency components.

18. A method according to claim 1 , wherein comparing the one or more primitives to one or more templates in a library of gesture templates comprises:
distorting at least a portion of a trajectory based at least in part upon frequency of motion components; and
searching for the distorted trajectory among the template(s) stored in the library.

19. A method according to claim 1 , wherein selecting from a result of the comparing a set of templates of possible gestures corresponding to the one or more primitives comprises:
determining a similarity between the one or more primitives and the set of templates by applying at least one similarity determiner; and
providing the similarity as an indication of quality of correspondence between the one or more primitives and the set of templates.

20. A method according to claim 19 , wherein determining a similarity between the one or more primitives and the set of templates by applying at least one similarity determiner comprises:
applying at least one of a correlation, a convolution, and a dot product.

21. A method according to claim 19 , wherein selecting from a result of the comparing a set of templates of possible gestures corresponding to the one or more primitives further comprises:
performing at least one of scaling and shifting to at least one of the one or more primitives and the set of templates.

22. A method according to claim 1 , wherein selecting from a result of the comparing a set of templates of possible gestures corresponding to the one or more primitives comprises:
disassembling at least a portion of a trajectory into a set of frequency components;
filtering set of frequency components to remove motions associated with jitter; and
searching for the set of filtered set of frequency components among the template(s) stored in the library.

23. A method according to claim 22 , wherein filtering set of frequency components comprises:
applying a Frenet-Serret filter.

24. The method according to claim 1 , wherein the at least one control object compr

Assignees

• Leap Motion Inc

Inventors

• Bedikian Raffi
• Marsden Jonathan
• Mertens Keith
• Holz David
• Sills Maxwell
• Perez Matias
• Hare Gabriel A
• Julian Ryan

Classifications

• G06F3/0325

  using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title