Using tracking to simulate direct tablet interaction in mixed reality

Patent metadata

Field	Value
Publication number	US-10445935-B2
Application number	US-201715607276-A
Country	US
Kind code	B2
Filing date	May 26, 2017
Priority date	May 26, 2017
Publication date	Oct 15, 2019
Grant date	Oct 15, 2019

Abstract

Optimizations are provided for facilitating interactions with virtual objects included within an augmented-reality scene. Initially, an augmented-reality scene is rendered for a user. Within that scene, an interactive virtual object of an application is rendered. Then, the position of the user's actual hand is determined relative to the interactive virtual object. When the user's actual hand is within a target threshold distance to the interactive virtual object, then a target visual cue is projected onto the interactive virtual object. When the user's actual hand is within an input threshold distance to the interactive virtual object, then an input visual cue is projected onto the interactive virtual object. Once the user's hand is within the input threshold distance to the interactive virtual object, then input may be provided to the application via the interactive object.

First claim

What is claimed is:

1. A computer system comprising:
one or more processor(s); and
one or more computer-readable hardware storage media having stored thereon computer-executable instructions that are executable by the one or more processor(s) to cause the computer system to:
on a wearable display of the computer system, render an augmented-reality scene for a user who is wearing the wearable display;
within the augmented-reality scene, render an interactive virtual object;
detect a position of a part of an actual hand of the user, the position being detected relative to a portion of the interactive virtual object such that the part of the user's actual hand is identified as approaching the portion of the interactive virtual object;
anticipate a type of gesture the user will subsequently use to interact with the interactive virtual object, wherein the anticipated gesture type is anticipated based at least partially on an identified type of the interactive virtual object, where the identified type is based on which interactive abilities the interactive virtual object is identified as having;
in response to determining that the part of the user's actual hand is within a target threshold distance to the portion of the interactive virtual object, display a target visual cue on the portion of the interactive virtual object;
in response to determining that the part of the user's actual hand is within an input threshold distance to the portion of the interactive virtual object, display an input visual cue on the portion of the interactive virtual object, the input visual cue indicating that input is now receivable for the interactive virtual object, the input being received at a same location as where the input visual cue is being displayed; and
upon a condition in which the part of the user's actual hand is simultaneously positioned within (i) either the target threshold distance or the input threshold distance to the portion of the interactive virtual object and (ii) either a corresponding target threshold distance or a corresponding input threshold distance to a corresponding different portion of a different interactive virtual object:
determine which interactive virtual object the user is attempting to interact with;
in response to determining that the user is attempting to interact with the corresponding different portion of the different interactive virtual object, display either the corresponding target visual cue or the corresponding input visual cue on the corresponding different portion of the different interactive virtual object; and
based on a subsequent movement of the part of the user's actual hand while the part of the user's actual hand is within the corresponding input threshold distance, provide input to an application via the different interactive virtual object.

2. The computer system of claim 1 , wherein the target visual cue is displayed at an anticipated ending location on the interactive virtual object where the part of the user's actual hand is anticipated to be moving towards.

3. The computer system of claim 1 , wherein the input visual cue displaces the target visual cue as soon as the part of the user's actual hand is within the input threshold distance.

4. The computer system of claim 1 , wherein, as the part of the user's actual hand progressively moves from the target threshold distance to the input threshold distance, the target visual cue progressively changes in appearance until the input visual cue is displayed.

5. The computer system of claim 1 , wherein the part of the user's actual hand is an actual finger of the user's actual hand.

6. The computer system of claim 1 , wherein execution of the computer-executable instructions further causes the computer system to:
detect that the part of the user's actual hand has passed through the interactive virtual object in the augmented-reality scene; and
in response to determining that the part of the user's actual hand exceeds the input threshold distance in relation to a back side of the portion of the interactive virtual object, stop providing input to the application via the interactive virtual object.

7. The computer system of claim 1 , wherein providing the input to the application via the interactive virtual object includes a scroll input gesture.

8. The computer system of claim 1 , wherein providing the input to the application via the interactive virtual object includes a zoom input gesture.

9. The computer system of claim 1 , wherein providing the input to the application via the interactive virtual object includes a tap input gesture.

10. The computer system of claim 1 , wherein providing the input to the application via the interactive virtual object includes a multiple touch input gesture.

11. A method implemented by one or more processor(s) of a computer system, the method comprising:
on a wearable display of the computer system, rendering an augmented-reality scene for a user who is wearing the wearable display;
within the augmented-reality scene, rendering an interactive virtual object;
detecting a position of a part of an actual hand of the user, the position being detected relative to a portion of the interactive virtual object such that the part of the user's actual hand is identified as approaching the portion of the interactive virtual object;
anticipate a type of gesture the user will subsequently use to interact with the interactive virtual object, wherein the anticipated gesture type is anticipated based at least partially on an identified type of the interactive virtual object, where the identified type is based on which interactive abilities the interactive virtual object is identified as having;
in response to determining that the part of the user's actual hand is within a target threshold distance to the portion of the interactive virtual object, displaying a target visual cue on the portion of the interactive virtual object;
in response to determining that the part of the user's actual hand is within an input threshold distance to the portion of the interactive virtual object, displaying an input visual cue on the portion of the interactive virtual object, the input visual cue indicating that input is now receivable for the interactive virtual object, the input being received at a same location as where the input visual cue is being displayed; and
upon a condition in which the part of the user's actual hand is simultaneously positioned within (i) either the target threshold distance or the input threshold distance to the portion of the interactive virtual object and (ii) either a corresponding target threshold distance or a corresponding input threshold distance to a corresponding different portion of a different interactive virtual object:
determine which interactive virtual object the user is attempting to interact with;
in response to determining that the user is attempting to interact with the corresponding different portion of the different interactive virtual object, display either the corresponding target visual cue or the corresponding input visual cue on the corresponding different portion of the different interactive virtual object; and
based on a subsequent movement of the part of the user's actual hand while the part of the user's actual hand is within the corresponding input threshold distance, providing input to an application via the different interactive virtual object.

12. The method of claim 11 , wherein the target threshold distance and the input threshold distance are dynamically adjustable based on a size of the interactive virtual object within the augmented-reality scene.

13. The method of claim 11 , wherein determi

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Schwarz Julia
• Calabrese David J
• Sheri Yasaman
• Witriol Daniel B

Classifications

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/01

  Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G06F2203/04806

  Zoom, i.e. interaction techniques or interactors for controlling the zooming operation · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title