Holographic keyboard display

Patent metadata

Field	Value
Publication number	US-9766806-B2
Application number	US-201414332334-A
Country	US
Kind code	B2
Filing date	Jul 15, 2014
Priority date	Jul 15, 2014
Publication date	Sep 19, 2017
Grant date	Sep 19, 2017

Abstract

Embodiments that relate to displaying holographic keyboard and hand images in a holographic environment are provided. In one embodiment depth information of an actual position of a user's hand is received. Using the depth information, a holographic hand image representing the user's hand is displayed in a virtual hand plane in the holographic environment. In response to receiving a keyboard activation input from the user and using the depth information, the holographic keyboard image is adaptively displayed in a virtual keyboard plane in the holographic environment at a virtual distance under the holographic hand image representing the user's hand.

First claim

The invention claimed is:

1. A method for displaying a holographic keyboard image and a holographic hand image representing a user's hand in a holographic environment, the method comprising:
receiving depth information of an actual position of the user's hand;
determining that the user's hand is spaced by an initial actual distance from a capture device that provides the depth information;
using the depth information, displaying the holographic hand image representing the user's hand in a virtual hand plane in the holographic environment;
receiving a keyboard activation input from the user;
in response to receiving the keyboard activation input, and using the depth information of the actual position of the user's hand, adaptively displaying in a virtual keyboard plane the holographic keyboard image spatially separated by a virtual distance from the holographic hand image representing the user's hand in the holographic environment;
determining that the user's hand moves to an updated actual distance from the capture device; and
in response to determining that the user's hand moves to the updated actual distance from the capture device, maintaining the holographic keyboard image spatially separated by substantially the virtual distance from the holographic hand image representing the user's hand.

2. The method of claim 1 , wherein the virtual hand plane of the holographic hand image forms an interaction angle with the virtual keyboard plane of the holographic keyboard image, the method further comprising:
determining that an initial actual plane of the user's hand changes by a rotation angle to an updated actual plane; and
in response to determining that the initial actual plane changes by the rotation angle to the updated actual plane, substantially maintaining the interaction angle between the virtual hand plane of the holographic hand image and the virtual keyboard plane of the holographic keyboard image.

3. The method of claim 1 , further comprising:
displaying a virtual element that accepts text input; and
wherein receiving the keyboard activation input further comprises determining that the user is gazing at the virtual element.

4. The method of claim 1 , wherein the keyboard activation input comprises audio input from the user.

5. The method of claim 1 , wherein the actual position of the user's hand is outside a display field of view of a display system that displays the holographic keyboard image and the holographic hand image, and the actual position of the user's hand is within a capture field of view of a capture device that provides the depth information.

6. The method of claim 1 , further comprising:
displaying one or more virtual shadows on the holographic keyboard image below the holographic hand image to provide a visual location cue of the virtual distance between the holographic hand image and the holographic keyboard image.

7. The method of claim 1 , further comprising:
determining that a holographic fingertip of the holographic hand image is located over a holographic key of the holographic keyboard image; and
in response to determining that the holographic fingertip is located over the holographic key, animating the holographic key to extend outwardly toward the holographic fingertip.

8. The method of claim 1 , wherein a holographic fingertip of the holographic hand image corresponds to a physical fingertip of the user's hand, the method further comprising:
determining that the holographic fingertip is located over a holographic key of the holographic keyboard image;
determining that the physical fingertip of the user's hand moves in a key-press direction by an actual key-press distance;
in response to determining that the physical fingertip moves in the key-press direction, animating the holographic fingertip and the holographic key to move toward the holographic keyboard by a virtual key-press distance that is less than the actual key-press distance to simulate friction between the holographic key and the holographic keyboard.

9. The method of claim 1 , further comprising:
determining that a holographic fingertip of the holographic hand image is located over a holographic key of the holographic keyboard image; and
in response to determining that the holographic fingertip of the holographic hand image is located over the holographic key of the holographic keyboard image, broadcasting one or more audio location cues.

10. A keyboard interface system for displaying a holographic keyboard image and a holographic hand image representing a user's hand in a holographic environment, the keyboard interface system comprising:
a display system;
a keyboard interface program executed by a processor of a computing device, the keyboard interface program configured to:
receive depth information of an actual position of the user's hand;
determine that the user's hand is spaced by an initial actual distance from a capture device that provides the depth information;
using the depth information, display via the display system the holographic hand image representing the user's hand in a virtual hand plane in the holographic environment;
receive a keyboard activation input from the user;
in response to receiving the keyboard activation input, and using the depth information of the actual position of the user's hand, adaptively display via the display system in a virtual keyboard plane the holographic keyboard image spatially separated by a virtual distance from the holographic hand image representing the user's hand in the holographic environment;
determine that the user's hand moves to an updated actual distance from the capture device; and
in response to determining that the user's hand moves to the updated actual distance from the capture device, maintain the holographic keyboard image spatially separated by substantially the virtual distance from the holographic hand image representing the user's hand.

11. The keyboard interface system of claim 10 , wherein the keyboard interface program is further configured to:
determine that the user's hand is spaced by an initial actual distance from a capture device that provides the depth information;
determine that the user's hand moves to an updated actual distance from the capture device; and
in response to determining that the user's hand moves to the updated actual distance from the capture device, maintain the holographic keyboard image spatially separated by substantially the virtual distance from the holographic hand image representing the user's hand.

12. The keyboard interface system of claim 10 , wherein the virtual hand plane of the holographic hand image forms an interaction angle with the virtual keyboard plane of the holographic keyboard image, and the keyboard interface program is further configured to:
determine that an initial actual plane of the user's hand changes by a rotation angle to an updated actual plane; and
in response to determining that the initial actual plane changes by the rotation angle to the updated actual plane, substantially maintain the interaction angle between the virtual hand plane of the holographic hand image and the virtual keyboard plane of the holographic keyboard image.

13. The keyboard interface system of claim 10 , wherein the keyboard interface program is further configured to:
display a virtual element that accepts text input; and
wherein receiving the keyboard activation input further comprises determining that the user is gazing at the virtual element.

14. The keyboard interface system of claim 10 , further comprising a capture device that captures the depth information, and wherein the actu

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bennet Rotem
• Geselowitz Lewey
• Zhang Wei
• Poulos Adam G
• Bevis John
• Pimmel Kim Pascal
• Fajt Nicholas Gervase

Classifications

• G03H2001/0061

  in haptic applications when the observer interacts with the holobject · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F1/1605

  Multimedia displays, e.g. with integrated or attached speakers, cameras, microphones · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G09G2370/24

  Keyboard-Video-Mouse [KVM] switch · CPC title