Generating virtual notation surfaces with gestures in an augmented and/or virtual reality environment

Patent metadata

Field	Value
Publication number	US-10147243-B2
Application number	US-201615369620-A
Country	US
Kind code	B2
Filing date	Dec 5, 2016
Priority date	Dec 5, 2016
Publication date	Dec 4, 2018
Grant date	Dec 4, 2018

Abstract

In an augmented reality and/or a virtual reality system, virtual annotation surfaces, or virtual sheets, or virtual whiteboards, may be materialized in response to a detected gesture. A user may annotate, adjust, store, review and revise the virtual annotation surfaces, and allow for collaboration with other users, while in the current virtual environment, and/or within another virtual environment, and/or outside of the virtual environment.

First claim

What is claimed is:

1. A method, comprising:
displaying, on a display of a head mounted display (HMD) device, a virtual environment;
detecting, outside of the HMD, a first gesture input based on a detected first position of a first handheld electronic device, and a detected first position of a second handheld electronic device;
generating, in response to detection of the first gesture input, a virtual notation surface;
detecting, outside of the HMD, a second gesture input based on a detected second position of the first handheld electronic device and a detected second position of the second handheld electronic device;
displaying, in response to detection of the second gesture input, the virtual notation surface as a first virtual object in the virtual environment;
detecting a release of an actuation of a manipulation device of at least one of the first handheld electronic device or the second handheld electronic device;
in response to the detected release of the actuation of the manipulation device, setting at least one of a size or a shape of the virtual notation surface; and
displaying the virtual notation surface as a second virtual object in the virtual environment.

2. The method of claim 1 , wherein the first handheld electronic device is operably coupled with the HMD, and the second handheld electronic device is operably coupled with the HMD.

3. The method of claim 1 , wherein setting the at least one of the size or the shape of the virtual notation surface includes:
setting the at least one of the size or the shape of the virtual notation surface based on the detected second position of the first handheld electronic device relative to the detected first position of the first handheld electronic device, and on the detected second position of the second handheld electronic device relative to the detected first position of the second handheld electronic device.

4. The method of claim 3 , wherein setting the at least one of the size or the shape of the virtual notation surface also includes:
setting the at least one of the size or the shape of the virtual notation surface in response to detection of the release of the actuation of the manipulation device, the second position of the first handheld electronic device and the detection of the second position of the second handheld electronic device being detected at a point corresponding to the detection of the release of the actuation of the manipulation device.

5. The method of claim 1 , further comprising:
detecting, outside of the HMD, a third gesture input;
annotating, in response to detection of the third gesture input, the virtual notation surface, the virtual notation surface, including:
detecting and tracking movement of a marking implement; and
annotating the virtual notation surface based on the detected movement of the marking implement; and

displaying the annotated virtual notation surface as the second virtual object in the virtual environment.

6. The method of claim 5 , annotating the virtual notation surface based on the detected movement of the marking implement including:
detecting actuation of a manipulation device of the marking implement;
initiating tracking of the movement of the marking implement in response to the detected actuation of the manipulation device;
detecting release of the actuation of the manipulation device; and
terminating tracking of the movement of the marking implement in response to the detected release of the manipulation device.

7. The method of claim 6 , annotating the virtual notation surface based on the detected movement of the marking implement also including:
detecting a contour corresponding to the tracked movement of the marking implement; and
annotating the virtual notation surface with a contour corresponding to the detected contour of the movement of the marking implement.

8. A system, comprising:
a computing device configured to generate a virtual reality environment, the computing device including:
a memory storing executable instructions; and
a processor configured to execute the instructions to cause the computing device to:
display, on a display of a head mounted display (HMD) device, a virtual environment;
detect, outside of the HMD, a first gesture input based on a detected first position of a first handheld electronic device, and a detected first position of a second handheld electronic device;
generate, in response to detection of the first gesture input, a virtual notation surface;
detect, outside of the HMD, a second gesture input based on a detected second position of the first handheld electronic device and a detected second position of the second handheld electronic device;
display, in response to detection of the second gesture input, the virtual notation surface as a first virtual object in the virtual environment;
detect a release of an actuation of a manipulation device of at least one of the first handheld electronic device or the second handheld electronic device;
in response to the detected release of the actuation of the manipulation device, set at least one of a size or a shape of the virtual notation surface; and
display the virtual notation surface as a second virtual object in the virtual environment.

9. The system of claim 8 , wherein the first handheld electronic device is operably coupled with the HMD, and the second handheld electronic device is operably coupled with the HMD.

10. The system of claim 8 , wherein, in setting the at least one of the size or the shape of the virtual notation surface, the instructions cause the computing device to:
set the at least one of the size or the shape of the virtual notation surface based on the detected second position of the first handheld electronic device relative to the detected first position of the first handheld electronic device, and on the detected second position of the second handheld electronic device relative to the detected first position of the second handheld electronic device.

11. The system of claim 10 , wherein, in setting the at least one of the size or the shape of the virtual notation surface, the instructions cause the computing device to:
set the at least one of the size or the shape of the virtual notation surface in response to detection of the release of the actuation of the manipulation device, the second position of the first handheld electronic device and the detection of the second position of the second handheld electronic device being detected at a point corresponding to the detection of the release of the actuation of the manipulation device.

12. The system of claim 8 , wherein the instructions cause the computing device to:
detect, outside of the HMD, a third gesture input;
annotate, in response to detection of the third gesture input, the virtual notation surface, including:
detect and track movement of a marking implement; and
annotate the virtual notation surface based on the detected movement of the marking implement; and

display the annotated virtual notation surface as the second virtual object in the virtual environment.

13. The system of claim 12 , wherein, in annotating the virtual notation surface based on the detected movement of the marking implement, the instructions cause the computing device to:
detect actuation of a manipulation device of the marking implement;
initiate tracking of the movement of the marking implement in response to the detected actuation of the manipulation device;
detect release of the actuation of the manipulation device; and
terminate tracking of the movement of the marking implement in response to the detected release of the manipulation device.

Assignees

• Google Inc
• Google Llc

Inventors

• Yamamoto Darwin
• Moore Evan
• Macgillivray Ian
• Bedard Jon

Classifications

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T2219/004

  Annotating, labelling · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title