Free-space user interface and control using virtual constructs

Patent metadata

Field	Value
Publication number	US-12405673-B2
Application number	US-202318506009-A
Country	US
Kind code	B2
Filing date	Nov 9, 2023
Priority date	Jan 15, 2013
Publication date	Sep 2, 2025
Grant date	Sep 2, 2025

Abstract

During control of a user interface via free-space motions of a hand or other suitable control object, switching between control modes can be facilitated by tracking the control object's movements relative to, and its penetration of, a virtual control construct (such as a virtual surface construct). The technology disclosed includes determining from the motion information whether a motion of the control object with respect to the virtual control construct is an engagement gesture, such as a virtual mouse click or other control device operation. The position of the virtual control construct can be updated, continuously or from time to time, based on the control object's location.

First claim

What is claimed is:

1. A computer-implemented method of controlling a machine user interface, comprising:
receiving information including motion information for a control object;
determining from the motion information whether a motion of the control object is an engagement gesture according to an occurrence of an engagement gesture applied to at least one virtual control construct defined within a field of view of an image capturing device;
determining a control to which the engagement gesture is applicable; and
manipulating the control according to at least the motion information.

2. The method of claim 1 , wherein determining from the motion information whether a motion of the control object is an engagement gesture according to an occurrence of an engagement gesture applied to a virtual control construct defined within a field of view of an image capturing device comprises:
determining whether an intersection occurred between control object and at least one virtual control construct.

3. The method of claim 2 , further comprising:
determining from the motion information whether the engagement includes continued motion after intersection.

4. The method of claim 1 , wherein determining from the motion information whether a motion of the control object is an engagement gesture according to an occurrence of an engagement gesture applied to a virtual control construct defined within a field of view of an image capturing device comprises:
determining whether a dis-intersection of the control object from the at least one virtual control construct occurred.

5. The method of claim 1 , wherein determining from the motion information whether a motion of the control object is an engagement gesture according to an occurrence of an engagement gesture applied to a virtual control construct defined within a field of view of an image capturing device comprises:
determining whether motion of the control object occurred relative to at least one virtual control construct.

6. The method of claim 1 , wherein determining from the motion information whether a motion of the control object is an engagement gesture according to an occurrence of an engagement gesture applied to a virtual control construct defined within a field of view of an image capturing device comprises:
determining from the motion information one or more engagement attributes defining an engagement gesture.

7. The method of claim 6 , wherein determining from the motion information one or more engagement attributes defining an engagement gesture includes:
determining a potential energy from the motion information.

8. The method of claim 1 , wherein determining from the motion information whether a motion of the control object is an engagement gesture according to an occurrence of an engagement gesture applied to a virtual control construct defined within a field of view of an image capturing device comprises:
identifying an engagement gesture by correlating motion information to at least one engagement gesture based at least upon one or more of:
motion of the control object, occurrence of any of an intersection, a dis-intersection or a non-intersection of the control object with the virtual control construct, and a set of engagement attributes.

9. The method of claim 1 , wherein determining a control to which the engagement gesture is applicable includes:
selecting at least one of a control associated with an application, a control associated with an operating environment, and a special control.

10. The method of claim 1 , wherein manipulating a control according to at least the motion information includes:
controlling a user interface in a first mode, and otherwise controlling the user interface in a second mode different from the first mode.

11. The method of claim 1 , further comprising:
updating at least a spatial position of the at least one virtual control construct based at least in part on a spatial position of the control object determined from the motion information; thereby enabling the spatial position of the at least one virtual control construct to follow tracked motions of the control object.

12. A system for controlling a machine user interface via free-space motions of a control object tracked with an image capturing device, the system comprising:
a processor; and
memory storing:
(i) motion information for the control object, and
(ii) processor-executable instructions for causing the processor to:
determine from the motion information whether a motion of the control object is an engagement gesture according to an occurrence of an engagement gesture applied to at least one virtual control construct defined within a field of view of the image capturing device,
determine a control to which the engagement gesture is applicable, and
manipulate the control according to at least the motion information.

13. A non-transitory machine-readable medium, storing one or more instructions which, when executed by one or more processors, cause the one or more processors to:
determine from motion information received for a control object whether a motion of the control object is an engagement gesture according to an occurrence of an engagement gesture applied to at least one virtual control construct defined within a field of view of an image capturing device;
determine a control to which the engagement gesture is applicable; and
manipulate the control according to at least the motion information.

14. The method of claim 1 , wherein the at least one virtual control construct includes at least one of:
virtual surface constructs, virtual linear constructs, virtual curvilinear constructs, virtual point constructs, or virtual solid constructs.

15. The method of claim 14 , wherein the virtual surface constructs comprise at least one of a plane, a curved open surface, a closed surface, a bounded open surface, and a predefined multi-dimensional virtual surface defined in two or three dimensions.

16. The method of claim 14 , wherein the virtual linear constructs comprise at least one of a one-dimensional virtual line, and a predefined line segment defined in one, two, or three dimensions.

17. The method of claim 14 , wherein the virtual curvilinear constructs comprise at least one of a curve, and a predefined curve segment defined in one, two, or three dimensions.

18. The method of claim 14 , wherein the virtual point constructs comprises a predefined zero-dimensional virtual point defined in one, two, or three dimensions.

19. The method of claim 14 , wherein the virtual solid constructs comprise at least one of a sphere, a cylinder, a cube, and a predefined three-dimensional virtual solid defined in three dimensions.

Assignees

• Ultrahaptics Ip Two Ltd

Inventors

• Bedikian Raffi
• Marsden Jonathan
• Mertens Keith
• Holz David

Classifications

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title