Augment orchestration in an artificial reality environment

Patent metadata

Field	Value
Publication number	US-11409405-B1
Application number	US-202017131563-A
Country	US
Kind code	B1
Filing date	Dec 22, 2020
Priority date	Dec 22, 2020
Publication date	Aug 9, 2022
Grant date	Aug 9, 2022

Abstract

Aspects of the present disclosure are directed to an artificial reality system orchestrating interactions between virtual object “augments.” The orchestration can include linking, which can be forming two or more augments into a combination, embedding an augment within an existing combination, or triggering an action mapped to the linking of those augments. Another type of orchestration can include extracting, which can refer to taking an augment out of an existing combination, either by removing it from the combination or copying the augment to leave a version in the combination and having another version outside the combination.

First claim

We claim:

1. A method for linking augments in a 3D artificial reality environment, wherein the 3D artificial reality environment orchestrates interactions between a plurality of augments, each of which is a virtual object with a 3D volume, and wherein each of the augments has one of a plurality of predefined types, the method comprising:
receiving an indication to link a first augment and a second augment of the plurality of augments,
wherein the first augment includes a first part, having a respective volume defined within the volume of the first augment, designated as a socket indicating that a first linking result for linking the first and second augments is to be automatically selected when the indication causes the second augment to be placed to intersect with the socket, and
wherein the first augment includes a second part, separate from the socket, indicating that a second linking result is to be automatically selected when the indication causes the second augment to be placed in association with the second part of the first augment;

identifying a first type of the plurality of predefined types for the first augment and a second type of the plurality of predefined types for the second augment;
obtaining a selected linking result by:
cross-referencing the first type for the first augment with the second type for the second augment,
wherein the 3D artificial reality environment makes a plurality of different results available for linking pairs of the plurality of augments based on predefined relationships between the respective types of the pairs of augments, and
wherein the selected linking result is one of multiple results including the first linking result and the second linking result available for linking of the first augment and the second augment, wherein the multiple linking results for linking the first augment and the second augment are based on one of the predefined relationships, defined between the first type for the first augment and the second type for the second augment; and

performing a disambiguation process comprising selecting between (1) the first linking result in response to determining that the indication to link the first augment causes the second augment to be placed to intersect with the socket and (2) the second linking result in response to determining that the indication to link the first augment causes the second augment to be placed in association with the second part of the first augment; and

implementing the selected linking result by:
when the first linking result is selected, automatically replacing at least one of the first augment and/or second augment with an augment of a resulting augment type specified by the first linking result; or
when the second linking result is selected, automatically performing an action, in relation to at least one of the first augment and/or second augment, specified by the second linking result, wherein the second linking result is implemented without further user input.

2. The method of claim 1 , wherein the indication to link the first augment and the second augment was received by an XR device recognizing a gesture, from a user, comprising the user causing the 3D volume of the first augment to intersect with the 3D volume of the second augment.

3. The method of claim 1 ,
wherein the selected linking result is the first linking result; and
wherein the first type for the first augment is an event type, the second type for the second augment is a person type, and the resulting augment type is an event type.

4. The method of claim 1 ,
wherein the selected linking result is the second linking result;
wherein the first type for the first augment is an event type, the second type for the second augment is a person type, and the action specified by the second linking result is sending a notification to a person associated with the second augment regarding an event associated with the first augment;
wherein the socket corresponds to a portion of the first augment for an invitee list; and
wherein the indication to link the first augment and the second augment was received by an XR device recognizing a gesture, from a user, comprising the user causing the 3D volume of the second augment to be placed in association with the second part of the first augment.

5. The method of claim 1 ,
wherein the selected linking result is the second linking result; and
wherein the first type for the first augment is an event type, the second type for the second augment is a person type, and the action specified by the second linking result is sending a notification to a person associated with the first augment regarding an event associated with the first augment.

6. The method of claim 1 , wherein the disambiguation process further comprises determining that the indication to link the first augment and the second augment included a gesture from a user, whereby the gestures caused the 3D volume of the first augment to intersect with the socket at least a threshold amount.

7. The method of claim 1 ,
wherein the selected linking result is the first linking result; and
wherein the method further comprises:
after the augment of the resulting augment type was added to the artificial reality environment, providing a set of one or more tools displayed in relation to the augment of the resulting augment type;
receiving a user interaction with a selected tool from the set of one or more tools; and
in response to the user interaction, creating a new augment with:
one or more properties based on the augment of the resulting augment type, and
a type defined in relation to the selected tool.

8. The method of claim 1 ,
wherein the selected linking result is the first linking result; and
wherein the method further comprises:
receiving a selection of one or more extraction augments from within the augment of the resulting augment type;
when a removal control has been activated, causing a version of the one or more extraction augments to be in the artificial reality environment while removing the one or more extraction augments from the augment of the resulting augment type; and
when no removal control has been activated, causing the version of the one or more extraction augments to be in the artificial reality environment while keeping another version of the one or more extraction augments in the augment of the resulting augment type.

9. A system for linking augments augments in a 3D artificial reality environment, wherein the 3D artificial reality environment orchestrates interactions between a plurality of augments, each of which is a virtual object with a 3D volume, and wherein each of the augments has one of a plurality of predefined types, the system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
receiving an indication to link a first augment and a second augment of the plurality of augments,
wherein the first augment includes a first part, having a respective volume defined within the volume of the first augment, designated as a socket indicating that a first linking result for linking the first and second augments is to be automatically selected when the indication causes the second augment to be placed to intersect with the socket, and
wherein the first augment includes a second part, separate from the socket, indicating that a second linking result is to be automatically selected when the indication causes the second augment to be placed in association with the second part of the first augment;

identifying a first type of the plurality of predefined types for the first augment and a second type of the

Assignees

• Facebook Tech Llc

Inventors

• Hlavac Michal
• Koga-Winn Benjamin Taylor
• Chak Wai Leong
• Stevens Jasper
• Louie Alexander Michael

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title