Predictive information for free space gesture control and communication

Patent metadata

Field	Value
Publication number	US-12086935-B2
Application number	US-202318373214-A
Country	US
Kind code	B2
Filing date	Sep 26, 2023
Priority date	Aug 29, 2013
Publication date	Sep 10, 2024
Grant date	Sep 10, 2024

Abstract

Free space machine interface and control can be facilitated by predictive entities useful in interpreting a control object's position and/or motion (including objects having one or more articulating members, i.e., humans and/or animals and/or machines). Predictive entities can be driven using motion information captured using image information or the equivalents. Predictive information can be improved applying techniques for correlating with information from observations.

First claim

What is claimed is:

1. A method of capturing gestural motion of a control object in a three-dimensional (3D) sensory space, the method including:
receiving observation information characterizing a surface of a control object from at least one image of a gestural motion of the control object in a three-dimensional (3D) sensory space;
obtaining a 3D model that represents the control object, the 3D model comprising one or more 3D subcomponents fit to the surface characterized,
wherein 3D subcomponents with least colliding subcomponents are fit to the surface;
wherein colliding subcomponents exhibit at least one subcomponent attribute incompatible with an attribute of another subcomponent; and
using the 3D model to facilitate control, communication and/or interaction with machines.

2. The method of claim 1 , further including adjusting the 3D model responsive to an error indication.

3. The method of claim 1 , further including adjusting the 3D model responsive to an error indication between a point on the surface and a corresponding point on the at least one of the 3D subcomponents, including:
pairing points on the surface with points on axes of the 3D subcomponents, wherein surface points lie on vectors that are normal to the axes; and
determining a reduced root mean squared deviation (RMSD) of distances between paired points.

4. The method of claim 1 , further including adjusting the 3D model responsive to an error indication between a point on the surface and a corresponding point on the at least one of the 3D subcomponents, including:
pairing points on the surface with points on the 3D subcomponents, wherein normal vectors to the points are parallel to each other; and
determining a reduced root mean squared deviation (RMSD) of distances between bases of the normal vectors.

5. The method of claim 1 , wherein fitting the one or more 3D subcomponents further includes fitting a set of closed curves to at least a portion of the surface.

6. The method of claim 5 , wherein closed curves include at least one of radial solids, capsuloids, spheres, ellipsoids, and hyperboloids.

7. The method of claim 5 , wherein at least one closed curve includes a contour and a surface defined by a set of points at a radius from the contour.

8. The method of claim 1 , wherein the control object is a hand and fitting the one or more 3D subcomponents further includes at least one of:
fitting capsuloids in finger portions of the surface; and
fitting radial solids in palm and/or wrist portions of the surface.

9. The method of claim 1 , wherein adjusting the 3D model further includes altering the 3D subcomponents to conform to at least one of length, width, orientation, and arrangement of portions of the surface.

10. The method of claim 9 , wherein altering the 3D subcomponents further includes applying a transformation matrix to a plurality of points on the 3D subcomponents.

11. The method of claim 9 , wherein altering the 3D subcomponents further includes determining a rotation matrix that provides a reduced root mean squared deviation (RMSD) between paired points on the surface and points on the 3D subcomponents.

12. The method of claim 1 , further including improving representation of the gestural motion by the 3D model by interpolating 3D model positions across time based on expected continuity in motion and deformation of the control object.

13. The method of claim 1 , wherein constructing the 3D model further includes selecting a pre-determined 3D model from an object library based on characteristic parameters of the control object.

14. The method of claim 1 , wherein constructing the 3D model further includes determining the 3D subcomponents from physical characteristics of a type of control object being observed; and wherein when the control object is a hand the physical characteristics of the hand include at least one of:
four fingers and a thumb of the hand;
a palm to which the fingers and the thumb are connected; and
positions and angles of the fingers and the thumb relative to each other and to the palm.

15. The method of claim 1 , wherein obtaining the 3D model further includes:
identifying one or more 3D subcomponents at an extremity of the control object; and
constraining error indication determination to subcomponents identified at an extremity.

16. The method of claim 1 , wherein obtaining the 3D model further includes:
linking the 3D subcomponents by representing a plurality of subcomponents using one or more artificial constructs; and
constraining error indication determination to the artificial constructs;
wherein the 3D subcomponents are fitted to a hand surface, further including at least one of:

using a line segment to represent finger subcomponents; and
using at least one of a line, square, circle, and ellipse to represent palm subcomponents.

17. The method of claim 1 , wherein obtaining the 3D model further includes:
defining a surface of a control object using a web of artificial constructs;
identifying surface portions of the control object using separate clusters of artificial constructs; and
constraining error indication determination to artificial constructs or clusters thereof.

18. The method of claim 1 , wherein obtaining the 3D model further includes improving representation of the gestural motion by the 3D model by:
detecting conflicting attributes between adjacent 3D subcomponents;
ranking the 3D subcomponents with conflicting attributes based on a degree of conflict;
selecting from ranked 3D subcomponents, 3D subcomponents with least conflicting attributes; and
fitting, to the surface, 3D subcomponents with least conflicting attributes.

19. The method of claim 1 , wherein obtaining the 3D model further includes:
obtaining gestural motion of the control object based on differences between 3D model positions across time, determined by repeatedly capturing observation information and obtaining an improved 3D model constructed using the observation information captures.

20. A non-transitory computer readable storage medium impressed with computer program instructions to capture gestural motion of a control object in a three-dimensional (3D) sensory space, which instructions, when executed on a processor, implement a method including:
receiving observation information characterizing a surface of a control object from at least one image of a gestural motion of the control object in a three-dimensional (3D) sensory space;
obtaining a 3D model that represents the control object, the 3D model comprising one or more 3D subcomponents fit to the surface characterized, wherein 3D subcomponents with least colliding subcomponents are fit to the surface; wherein colliding subcomponents exhibit at least one subcomponent attribute incompatible with an attribute of another subcomponent; and
using the 3D model to facilitate control, communication and/or interaction with machines.

21. A system for capturing gestural motion of a control object in a three-dimensional (3D) sensory space, comprising:
a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to:
receive observation information characterizing a surface of a control object from at least one image of a gestural motion of the control object in a three-dimensional (3D) sensory space;
obtain a 3D model that represents the control object, the 3D model comprising one or more 3D subcomponents fit to the surface characterized, wherein
3

Assignees

• Ultrahaptics Ip Two Ltd

Inventors

• Horowitz Kevin A
• Holz David S

Classifications

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• H04N13/275

  from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals · CPC title

• A61B2017/00207

  with hand gesture control or hand gesture recognition · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06T2200/08

  involving all processing steps from image acquisition to 3D model generation · CPC title