Extended reality virtual assistant

Patent metadata

Field	Value
Publication number	US-10304239-B2
Application number	US-201715655762-A
Country	US
Kind code	B2
Filing date	Jul 20, 2017
Priority date	Jul 20, 2017
Publication date	May 28, 2019
Grant date	May 28, 2019

Abstract

Methods, devices, and apparatuses are provided to facilitate a positioning of an item of virtual content in an extended reality environment. For example, a first user may access the extended reality environment through a display of a mobile device, and in some examples, the methods may determine positions and orientations of the first user and a second user within the extended reality environment. The methods may also determine a position for placement of the item of virtual content in the extended reality environment based on the determined positions and orientations of the first user and the second user, and perform operations that insert the item of virtual content into the extended reality environment at the determined placement position.

First claim

We claim:

1. A computer-implemented extended reality method, comprising:
obtaining, by one or more processors of a computing system, a position and an orientation of a first user in an extended reality environment, the computing system being in communication with one or more of a first device of the first user and a second device of a second user across a communications network;
obtaining, by the one or more processors of the computing system, a position and an orientation of the second user in the extended reality environment;
determining, by the one or more processors of the computing system, one or more placement positions for an item of virtual content in the extended reality environment at least partially based on the positions and orientations of the first user and the second user, wherein a first placement position associated with the first device is based at least on the position and the orientation of the first user, and wherein a second placement position associated with the second device is based at least on the position and the orientation of the second user; and
transmitting, by the one or more processors of the computing system, the first placement position or the second placement position to one or more of the first device and the second device across the communications network, wherein the item of virtual content is inserted into the extended reality environment at a placement position determined from the first placement position or the second placement position.

2. The computer-implemented extended reality method of claim 1 , wherein:
the computing system comprises the first device of the first user;
obtaining the position and the orientation of the second user comprises receiving the position and the orientation of the second user; and
the method further comprising displaying, on the first device, the item of virtual content within the extended reality environment at the placement position.

3. The computer-implemented extended reality method of claim 2 , wherein:
the first device of the first user comprises a display;
the method further comprises displaying digital content that establishes the extended reality environment on the display; and
displaying the item of virtual content within the extended reality environment comprises displaying the item of virtual content on the display at the placement position.

4. The computer-implemented extended reality method of claim 1 , wherein:
the orientation of the first user comprises one or more of an orientation of at least a portion of a body of the first user and an orientation of the first device of the first user; and
the orientation of the second user comprises one or more of an orientation of at least a portion of a body of the second user and an orientation of the second device of the second user.

5. The computer-implemented extended reality method of claim 1 , wherein the item of virtual content comprises a virtual assistant, the virtual assistant comprising one or more of an avatar and an item of animated digital content.

6. The computer-implemented extended reality method of claim 1 , wherein determining the one or more placement positions for the item of virtual content in the extended reality environment comprises:
identifying a plurality of candidate positions for placement of the item of virtual content within the extended reality environment;
computing a respective placement score for each of the plurality of candidate positions based on the positions and orientations of the first user and the second user; and
based on the respective placement score for each of the plurality of candidate positions, establishing one of the plurality of candidate positions as the first placement position or the second placement position.

7. The computer-implemented extended reality method of claim 6 , further comprising:
obtaining data that characterizes an object disposed within the extended reality environment, the obtained data comprising a type of the object and one or more of a position and a dimension of the object within the extended reality environment; and
computing the respective placement score for each of the plurality of candidate positions based on the positions and orientations of the first user and the second user, and based on one or more of the type, the position, and the dimension of the object within the extended reality environment.

8. The computer-implemented extended reality method of claim 6 , wherein computing the respective placement score comprises computing, for a corresponding one of the plurality of candidate positions, the respective placement score based on: displacements between the corresponding one of the plurality of candidate positions and the positions of the first user and the second user within the extended reality environment.

9. The computer-implemented extended reality method of claim 6 , wherein computing the respective placement score is further based on one or more of: viewing angles of the item of virtual content relative to the orientations of the first and second users; and displacements between a corresponding one of the plurality of candidate positions and the position of the first user and the position of the second user within the extended reality environment.

10. The computer-implemented extended reality method of claim 6 , further comprising:
obtaining depth map data characterizing a depth map of a visible portion of the extended reality environment that is visible to the first user;
obtaining semantic data indicative of an output of a semantic scene analysis of the visible portion of the extended reality environment, the semantic data identifying one or more of a type, a position, and a dimension of a physical object disposed within the visible portion of the extended reality environment; and
identifying the plurality of candidate positions for placement of the item of virtual content within the extended reality environment based on one or more of the obtained depth map data and the obtained semantic data.

11. The computer-implemented extended reality method of claim 1 , further comprising:
detecting a gestural input of the first user; and
performing operations that modify the placement position of the item of virtual content within the extended reality environment based on the detected gestural input.

12. The computer-implemented extended reality method of claim 1 , further comprising:
detecting a gestural input of the first user;
in response to the detected gestural input, projecting a gestural vector onto a depth map of a visible portion of the extended reality environment that is visible to the first user;
identifying an object associated with the projected gestural vector; and
performing operations that present digital content associated with the identified object to the first user.

13. The computer-implemented extended reality method of claim 1 , further comprising:
detecting one or more of a gestural input and a spoken input of the first user, the one or more of the gestural input and the spoken input requesting a performance of an action by the item of virtual content;
identifying an action requested by the first user based on the one or more of the gestural input and the spoken input; and
performing operations consistent with the identified action.

14. The computer-implemented extended reality method of claim 1 , wherein the computing system is in communication with the first device and the second device.

15. The computer-implemented extended reality method of claim 1 , wherein the computing system comprises the second device.

16. An apparat

Assignees

• Qualcomm Inc

Inventors

• Gorur Sheshagiri Pushkar
• Baheti Pawan Kumar
• Gupte Ajit Deepak
• Lakshmikantha Sandeep Kanakapura

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title