Artificial reality application lifecycle

Patent metadata

Field	Value
Publication number	US-11762952-B2
Application number	US-202117360836-A
Country	US
Kind code	B2
Filing date	Jun 28, 2021
Priority date	Jun 28, 2021
Publication date	Sep 19, 2023
Grant date	Sep 19, 2023

Abstract

Aspects of the present disclosure are directed to an artificial reality (XR) application system controlling applications in an artificial reality environment. In various cases, these controls include automatically suggesting XR applications by determining an XR context and identifying applications that match the XR context. These applications can be suggested to a user, who can authorize their execution, setting permissions for the application. In some cases, applications can be divided into components which can be progressively downloaded. By providing application suggestions relevant to the current context and progressively downloading application components, applications can appear ambient, rather than relying on users to constantly download, install, or activate applications. Permissions for applications may be revoked permanently or for certain situations—either through user permissions selections or automatically in response to determined user intents. When multiple applications are simultaneously authorized to execute, the XR application system can employ a ranking system to prevent overcrowding.

First claim

We claim:

1. A method for suggesting, authorizing, and controlling application output in an artificial reality (XR) device, the method comprising:
identifying one or more artificial reality environment anchor points, which specify real-world locations, to which application output will be attached in an artificial reality environment;
determining a plurality of features of a current XR context, defining a) characteristics of a user and b) an automatically determined state of the real-world in a vicinity of the user, the current XR context comprising one or more of: determinations based on sensor data gathered by the XR device, data retrieved from a third party source, data local to the XR device, or any combination thereof;
providing suggestions for one or more applications that match the current XR context and at least one anchor point of the one or more artificial reality environment anchor points, wherein each particular application is identified by:
determining that a) a threshold subset of multiple features from the plurality of features of the current XR context have been mapped to the particular application and that b) the particular application is configured to provide output in relation to the at least one anchor point; or
applying a machine learning model, trained to determine a match score, to multiple features of the current XR context and to a type specified for the particular application, and determining that a resulting match score is above a threshold;

receiving a user action to authorize a selected application of the suggested one or more applications;
in response to the user action:
obtaining components of the selected application; and
setting permissions for the selected application that allow the selected application to provide output, in the artificial reality environment, attached to the at least one anchor point; and

identifying a user intent by identifying that a user ignored or closed a threshold amount of outputs from the selected application within a time window; and
in response to identifying the user intent, removing one or more of the permissions set for the selected application.

2. The method of claim 1 , wherein each of the one or more anchor points corresponds to a respective particular real-world object identified by the XR device.

3. The method of claim 1 , wherein the current XR context comprises:
one or more identified objects in the artificial reality environment,
current location data for the XR device,
one or more aspects of a user state and/or one or more relationships to one or more other users identified in the area of the XR device, and
environment conditions.

4. The method of claim 1 , wherein identifying the one or more applications is performed by the determining that the threshold subset of features from the current XR context has been mapped to the particular application.

5. The method of claim 1 , wherein identifying the one or more applications is performed by the applying the machine learning model.

6. The method of claim 1 , wherein identifying each particular application further comprises:
determining a category, of multiple categories, for each of one or more potential applications; and
adding, to the suggestions for one or more applications, a maximum amount of the one or more potential applications in one of the multiple categories, wherein at least one of the one or more potential applications is excluded from the suggestions for one or more applications due to the maximum amount already having been reached for the category of that application.

7. The method of claim 1 , wherein the received user action includes identifying a user hand gesture that indicates the selected application, from the suggestions, and drags an indication of the selected application to another location in the artificial reality environment.

8. The method of claim 1 , wherein the obtaining components of the selected application comprises:
determining, based on which aspects of the selected application will be used to provide the output in the artificial reality environment, a priority order for the components of the selected application; and
scheduling download of the components of the selected application according to the priority order.

9. The method of claim 1 , wherein the permissions set for the selected application include two or more of:
permissions for creating augments in the artificial reality environment,
permissions to provide notifications,
permissions to access input signals,
permissions to receive context features,
permissions to write into a shared XR space,
permissions to exchange objects or other data with other authorized applications, or
any combination thereof.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for suggesting, authorizing, and controlling application output in an artificial reality (XR) device, the process comprising:
determining a current XR context comprising one or more of: determinations based on sensor data, data retrieved from a third party source, data local to the XR device, or any combination thereof;
providing suggestions for one or more applications that match the current XR context, wherein each particular application is identified by:
determining that a threshold subset of features from the current XR context have been mapped to the particular application; or
applying a machine learning model, trained to determine a match score, to multiple features of the current XR context and to a type specified for the particular application, and determining that a resulting match score is above a threshold;

receiving a user action to authorize a selected application of the suggested one or more applications;
in response to the user action, setting permissions for the selected application that allow the selected application to provide output;
identifying a user intent by identifying that a user ignored or closed a threshold amount of outputs from the selected application; and
in response to identifying the user intent, removing one or more of the permissions set for the selected application.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the process further comprises obtaining components of the selected application by:
determining, based on which aspects of the selected application will be used to provide the output, a priority order for the components of the selected application; and
scheduling download of the components of the selected application according to the priority order.

12. The non-transitory computer-readable storage medium of claim 10 , wherein the current XR context comprises:
one or more identified objects,
current location data for the XR device, and
calendar items for a current user.

13. The non-transitory computer-readable storage medium of claim 10 , wherein identifying the one or more applications is performed by the determining that the threshold subset of features from the current XR context have been mapped to the particular application.

14. The non-transitory computer-readable storage medium of claim 10 , wherein identifying the one or more applications is performed by the applying the machine learning model.

15. A computing system for suggesting, authorizing, and controlling application output in an artificial reality (XR) device, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a pr

Assignees

• Meta Platforms Tech Llc

Inventors

• Hlavac Michal
• Stevens Jasper
• Zwiegincew Arthur
• Louie Alexander Michael

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F18/2178 Primary

  based on feedback of a supervisor · CPC title

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• H04M1/72454

  according to context-related or environment-related conditions · CPC title

• H04M1/724097

  Worn on the head · CPC title