Hand motion and orientation-aware buttons and grabbable objects in mixed reality

Patent metadata

Field	Value
Publication number	US-11320911-B2
Application number	US-201916296833-A
Country	US
Kind code	B2
Filing date	Mar 8, 2019
Priority date	Jan 11, 2019
Publication date	May 3, 2022
Grant date	May 3, 2022

Abstract

Systems and methods are provided for detecting user-object interaction in mixed-reality environments. A mixed-reality system detects a controller gesture with an associated controller orientation in the mixed-reality environment. The mixed-reality system then determines an interaction region for the controller gesture and identifies one or more virtual objects within the interaction region. The virtual objects each have an associated orientation affinity. Subsequently, the mixed-reality system determines an orientation similarity score between the controller orientation and the orientation affinity for each virtual object within the interaction region. In response to determining that at least one orientation similarity score exceeds a predetermined threshold, the mixed-reality system executes an interaction between the controller and the virtual object that has the greatest orientation similarity score.

First claim

What is claimed is:

1. A system for detecting user-object interaction in a mixed-reality environment, the system comprising:
one or more processors; and
one or more computer-readable media having stored computer-executable instructions that are operable, when executed by the one or more processors, to cause the system to at least:
detect a physical controller gesture of a physical controller comprising a physical object in the mixed-reality environment, wherein the physical controller gesture has an associated physical controller orientation identifying a virtual object interaction orientation for the physical controller that is based on a physical geometry of the physical object that constrains a way in which the physical object can be interacted with;
determine an interaction region for the physical controller gesture;
identify a plurality of virtual objects within the interaction region, wherein each of the plurality of virtual objects has an associated orientation affinity defining a preferred interaction direction for the virtual object, conformance of the associated physical controller orientation to the associated orientation affinity enabling the physical controller to interact with the virtual object;
determine an orientation similarity score between the associated physical controller orientation and the associated orientation affinity for each of the plurality of virtual objects within the interaction region;
in response to determining that at least one orientation similarity score exceeds a predetermined threshold, select a particular virtual object of the plurality of virtual objects that has a greatest orientation similarity score out of all orientation similarity scores determined for each of the plurality of virtual objects; and
based at least on selecting the particular virtual object, execute an interaction between the physical controller and the particular virtual object.

2. The system of claim 1 , wherein each orientation similarity score is based on a dot product of the associated physical controller orientation and the associated orientation affinity for one virtual object of the plurality of virtual objects.

3. The system of claim 1 , wherein the computer-executable instructions are further operable to cause the system to:
determine a directional component of the physical controller gesture; and
determine a directional preference for at least one of the plurality of virtual objects within the interaction region.

4. The system of claim 3 , wherein the interaction between the physical controller and the particular virtual object is executed only upon determining that the directional component of the physical controller gesture complies with the directional preference for at least one of the plurality of virtual objects.

5. The system of claim 3 , wherein the directional component of the physical controller gesture is based on a velocity of the physical controller gesture.

6. The system of claim 1 , wherein the interaction between the physical controller and the particular virtual object is executed only upon determining that a pose of the physical controller corresponds to an interactive pose.

7. The system of claim 1 , wherein the interaction region for the physical controller gesture is determined based on a pose of the physical controller.

8. The system of claim 1 , wherein the interaction region for the physical controller gesture is a conical region extending away from the physical controller.

9. The system of claim 1 , wherein the particular virtual object comprises an affordance of a hologram in the mixed-reality environment.

10. A method for detecting user-object interaction in a mixed-reality environment, the method comprising:
detecting a physical controller gesture of a physical controller comprising a physical object in the mixed-reality environment, wherein the physical controller gesture has an associated physical controller orientation identifying a virtual object interaction orientation for the physical controller that is based on a physical geometry of the physical object that constrains a way in which the physical object can be interacted with;
determining an interaction region for the physical controller gesture;
identifying one or more virtual objects within the interaction region, wherein each of the one or more virtual objects has an associated orientation affinity defining a preferred interaction direction for the virtual object, conformance of the associated physical controller orientation to the associated orientation affinity enabling the physical controller to interact with the virtual object, at least a particular virtual object of the one or more virtual objects having a plurality of orientation affinities;
determining a particular orientation affinity of the particular virtual object based on a determined pose of the physical controller gesture, wherein different poses correspond to different orientation affinities;
determining an orientation similarity score between the associated physical controller orientation and the particular orientation affinity;
in response to determining that the orientation similarity score for the particular orientation affinity and the associated physical controller orientation exceeds a predetermined threshold, selecting the particular virtual object; and
based at least on selecting the particular virtual object, execute an interaction between the physical controller and the particular virtual object.

11. The method of claim 10 , wherein the orientation similarity score is based on a dot product of the associated physical controller orientation and the associated orientation affinity for the particular virtual object.

12. The method of claim 10 , further comprising:
determining a directional component of the physical controller gesture; and
determining a directional preference for the particular virtual object.

13. The method of claim 12 , wherein the interaction between the physical controller and the particular virtual object is executed only upon determining that the directional component of the physical controller gesture complies with the directional preference for the particular virtual object.

14. The method of claim 12 , wherein the directional component of the physical controller gesture is based on a velocity of the physical controller gesture.

15. The method of claim 10 , wherein the interaction between the physical controller and the particular virtual object is executed only upon determining that the determined pose of the physical controller corresponds to an interactive pose.

16. The method of claim 10 , wherein the interaction region for the physical controller gesture is determined based on the determined pose of the physical controller.

17. The method of claim 10 , wherein the interaction region for the physical controller gesture is a conical region extending away from the physical controller.

18. The method of claim 10 , wherein the particular virtual object comprises an affordance of a hologram in the mixed-reality environment.

19. One or more hardware storage devices having stored thereon computer-executable instructions, the computer-executable instructions being executable by one or more processors of a computer system to cause the computer system to:
detect a physical controller gesture of a physical controller comprising a physical object in a mixed-reality environment, wherein the physical controller gesture has an associated physical controller orientation identifying a virtual object interaction orientation for the

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Schwarz Julia
• Ray Jason Michael
• Meekhof Casey Leon

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title