Near interaction mode for far virtual object

Patent metadata

Field	Value
Publication number	US-11320957-B2
Application number	US-201916363684-A
Country	US
Kind code	B2
Filing date	Mar 25, 2019
Priority date	Jan 11, 2019
Publication date	May 3, 2022
Grant date	May 3, 2022

Abstract

A computing system is provided. The computing system includes a head mounted display (HMD) device including a display, a processor configured to execute one or more programs, and associated memory. The processor is configured to display a virtual object at least partially within a field of view of a user on the display, identify a plurality of control points associated with the virtual object, and determine that one or more of the control points associated with the virtual object are further than a predetermined threshold distance from the user. The processor is configured to, based on the determination, invoke a far interaction mode for the virtual object and receive a trigger input from the user. In response to the trigger input in the far interaction mode, the processor is configured to invoke a near interaction mode and display a virtual interaction object within the predetermined threshold distance from the user.

First claim

The invention claimed is:

1. A computing system, comprising:
a head mounted display (HMD) device including a display;
a processor and associated memory, the processor being configured to execute one or more programs stored in the memory to:
display a virtual object at least partially within a field of view of a user on the display;
identify a first plurality of control points associated with the virtual object;
determine that one or more of the first plurality of control points associated with the virtual object are further than a predetermined threshold distance from the user;
based on the determination, invoke a far interaction mode for the virtual object;
in the far interaction mode, receive a trigger input from the user;
in response to receiving the trigger input in the far interaction mode, invoke a near interaction mode and display a virtual interaction object and a second plurality of control points,
wherein the virtual interaction object and the second plurality of control points are displayed within the predetermined threshold distance from the user concurrently with display of the virtual object, and
wherein the second plurality of control points are associated with the virtual interaction object;

receive a user interaction input performed by the user at one or more of the second plurality of control points associated with the virtual interaction object; and
execute control of the virtual object as specified by the user interaction input at the one or more of the second plurality of control points associated with the virtual interaction object.

2. The system of claim 1 , the processor further configured to:
in the far interaction mode, prior to receiving the trigger input from the user:
generate a virtual ray from a hand of the user, the virtual ray locked with respect to movement of the hand of the user;
in response to the virtual ray intersecting the virtual object, receive a user selection of the virtual object, the virtual object being in the far interaction mode; and
via the virtual ray, execute control of the virtual object according to gestures of the hand of the user,
wherein the trigger input is executed by the user with respect to the virtual ray to invoke the near interaction mode and display the virtual interaction object within the predetermined threshold distance from the user.

3. The system of claim 1 , wherein the virtual interaction object is one of a plurality of virtual interaction objects that the processor is configured to generate, and
the trigger input is one of a plurality of possible trigger inputs, each of the possible trigger inputs being a corresponding gesture, each gesture being associated with one virtual interaction object of the plurality of virtual interaction objects.

4. The system of claim 1 , wherein the virtual interaction object is one of a plurality of virtual interaction objects that the processor is configured to generate, and
the plurality of virtual interaction objects includes at least one of a pinchable object, handles associated with the virtual object, at least one virtual button, a bounding box, a control widget, and a context menu.

5. The system of claim 1 , wherein the virtual object extends beyond the user's field of view and at least one of the first plurality of control points associated with the virtual object are outside of the user's field of view.

6. The system of claim 1 , wherein the virtual interaction object displayed in the near interaction mode is a resized copy of the virtual object and all of the second plurality of control points are within the user's field of view and within the predetermined threshold distance.

7. The system of claim 1 , wherein the trigger input includes at least one of a predefined gesture, a vocal command, a gaze direction, and eye movement.

8. The system of claim 1 , wherein the virtual interaction object displayed within the predetermined threshold distance from the user is locked in location with respect to the user as the user moves.

9. The system of claim 1 , wherein the predetermined threshold distance from the user is in a range of 460 mm to 580 mm.

10. A method for use with a head mounted display (HMD) device including a processor and a display, comprising:
displaying a virtual object at least partially within a field of view of a user on the display;
identifying a first plurality of control points associated with the virtual object;
determining that one or more of the first plurality of control points associated with the virtual object are further than a predetermined threshold distance from the user;
based on the determination, invoking a far interaction mode for the virtual object;
in the far interaction mode, receiving a trigger input from the user;
in response to receiving the trigger input in the far interaction mode, invoking a near interaction mode and displaying a virtual interaction object and a second plurality of control points,
wherein the virtual interaction object and the second plurality of control points are displayed within the predetermined threshold distance from the user concurrently with display of the virtual object, and
wherein the second plurality of control points are associated with the virtual interaction object;

receiving a user interaction input performed by the user at one or more of the second plurality of control points associated with the virtual interaction object; and
executing control of the virtual object as specified by the user interaction input at the one or more of the second plurality of control points associated with the virtual interaction object.

11. The method of claim 10 , further comprising, at the processor:
in the far interaction mode, prior to receiving the trigger input from the user:
generating a virtual ray from a hand of the user, the virtual ray locked with respect to movement of the hand of the user;
in response to the virtual ray intersecting the virtual object, receiving a user selection of the virtual object; and
via the virtual ray, executing control of the virtual object according to gestures of the hand of the user,
wherein the trigger input is executed by the user with respect to the virtual ray to invoke the near interaction mode and display the virtual interaction object within the predetermined threshold distance from the user.

12. The method of claim 10 , wherein the virtual interaction object is one of a plurality of virtual interaction objects that is generated, and
the trigger input is one of a plurality of possible trigger inputs, each of the possible trigger inputs being a corresponding gesture, each gesture being associated with one virtual interaction object of the plurality of virtual interaction objects.

13. The method of claim 10 , wherein the virtual interaction object is one of a plurality of virtual interaction objects that is generated, and
the plurality of virtual interaction objects includes at least one of a pinchable object, handles associated with the virtual object, at least one virtual button, a bounding box, a control widget, and a context menu.

14. The method of claim 10 , wherein the virtual object extends beyond the user's field of view and at least one of the first plurality of control points associated with the virtual object are outside of the user's field of view.

15. The method of claim 10 , wherein the virtual interaction object displayed in the near interaction mode is a resized copy of the virtual object and all of the second plurality of control points are within the user's field of view and within the predetermined threshold distance.

16.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Tang sheng kai
• Schwarz Julia
• Gable Thomas Matthew
• Meekhof Casey Leon
• Sharkasi Nahil Tawfik
• Kamuda Nicholas Ferianc
• Neff Joshua Kyle
• Kwok Alton

Classifications

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title