Sharing of resources for generating augmented reality effects

Patent metadata

Field	Value
Publication number	US-12026802-B2
Application number	US-202217592052-A
Country	US
Kind code	B2
Filing date	Feb 3, 2022
Priority date	Jun 25, 2020
Publication date	Jul 2, 2024
Grant date	Jul 2, 2024

Abstract

An augmented reality (AR) effect system can improve application of AR effects by sharing resources between AR effects. The AR effect system can employ manifests for AR effects that define which resources are required to render each AR effect. The AR effect system can organize rendering operations used by selected AR effects into a pipeline and can use the manifests of the AR effects to determine when each resource will be needed. Based on this pipeline, the AR effect system can create a cache order defining a resource schedule which specifies, when a resource is freed, conditions for whether to save the resource to a local cache or unload the resource. As rendering of the video with the AR effects progresses, the resource schedule can control whether resources not currently being used to render an AR effect should be unloaded or cached for fast access for future render operations.

First claim

We claim:

1. A method for pre-storing resources for augmented reality (AR) effects, the method comprising:
predictively pre-storing one or more resources for one or more AR effects by:
generating a prediction of reduced loading time during rendering of one or more AR effects resulting from pre-loading of a set of the one or more resources, wherein the prediction is based on—AR effect statistics of past frequencies of AR effect use, and
resource characteristics for resource sizes and/or expected latency in accessing resources; and

in response to the prediction, adding, to a local cache, the set of one or more resources;

receiving an indication of a video associated with the one or more AR effects;
causing one or more frames of the video to be rendered, with the one or more AR effects,
wherein the rendering with the one or more AR effects uses the pre-loaded set of one or more resources; and
wherein the rendering using the pre-loaded set of one or more resources is performed more quickly than rendering the one or more AR effects when the set of one or more resources is not available in the local cache;

determining that at least one particular resource has been freed;
determining, based on a resource schedule, that the freed resource should be unloaded; and
in response, unloading the freed resource from the local cache.

2. The method of claim 1 , wherein at least some of the AR effect statistics are identified as being specific to a context corresponding to a current situation, the context identifying one or more of: user characteristics, a location, identified content of the video, or any combination thereof.

3. The method of claim 1 , wherein at least some of the AR effect statistics are identified as being specific to a current user, signifying:
which AR effects the current user uses most often; and/or
in what context the current user selects AR effects.

4. The method of claim 1 , wherein at least some of the AR effect statistics are identified as being specific to a current location, signifying AR effects most often selected for video captured at the current location.

5. The method of claim 1 , wherein the generating the prediction of reduced loading time is based on processing by a machine learning model trained to predict which AR effects a user will select based on at least the AR effect statistics of past frequencies of AR effect use, user demographics, and location data.

6. The method of claim 1 , wherein the generating the prediction of reduced loading time is based on processing by a machine learning model trained to predict which AR effects a user will select based on at least the AR effect statistics of past frequencies of AR effect use.

7. The method of claim 1 , wherein the predictively pre-storing the one or more resources further includes determining that one or more conditions, for storing the one or more resources in the local cache, was evaluated by making a comparison of a) a resource score, computed for the one or more resources, with b) a cache threshold; wherein the resource score is based on the AR effect statistics of past frequencies of AR effect use.

8. The method of claim 1 , wherein the one or more frames of the video is to be rendered with an AR effect, of the one or more AR effects, as a result of one or more voice commands from a user, wherein receiving the one or more voice commands comprised:
identifying a spoken activation phrase;
receiving a particular voice command;
selecting, using a pre-established mapping of commands to AR effects, the AR effect by mapping a version of the received particular voice command to the AR effect; and
providing an indication of the selected AR effect.

9. The method of claim 1 , wherein the local cache is in RAM or flash storage.

10. The method of claim 1 , wherein the prediction is further based on determined content of the video and an identification of AR effects most often selected for video with that content.

11. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for pre-storing resources for augmented reality (AR) effects, the process comprising:
receiving an indication of a first video associated with one or more AR effects;
causing one or more first frames of the first video to be rendered, wherein the rendering uses one or more resources for the one or more AR effects;
generating a resource score for the one or more resources, wherein the resource score is based on:
AR effect statistics of past frequencies of AR effect use, and/or
determined content of the first video and an identification of AR effects most often selected for videos with the determined content;

determining that the resource score for the one or more resources is above a threshold and, in response, locally storing the one or more resources; and
causing one or more second frames of the first video and/or one or more frames of a second video to be rendered using the locally stored one or more resources, wherein the rendering is configured to retrieve the one or more resources from a remote source when the one or more resources are not available locally.

12. The computer-readable storage medium of claim 11 , wherein the resource score is based on the AR effect statistics of past frequencies of AR effect use through a comparison of (a) a total expected load time savings by caching the one or more resources with (b) an expected cost of caching the one or more resources.

13. The computer-readable storage medium of claim 12 , wherein the total expected load time savings is be based on an estimated cost of reloading the one or more resources and an estimated frequency of further use of the one or more resources in further rendering.

14. The computer-readable storage medium of claim 13 , wherein the estimated frequency of further use of the one or more resources is based on determined video content and statistics for which AR effects are most often selected for that content.

15. The computer-readable storage medium of claim 11 , wherein the determined content of the first video is based on determinations of one or more of: identified objects or locations depicted, a video size, a video encoding, a video framerate, or any combination thereof.

16. The computer-readable storage medium of claim 11 , wherein the threshold is a dynamic value that changes based on remaining space in a cache.

17. A computing system for pre-storing resources for augmented reality (AR) effects, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
predictively pre-storing one or more resources for one or more AR effects by:
generating a prediction of reduced loading time during rendering of one or more AR effects resulting from pre-loading of a set of the one or more resources; and
in response to the prediction, adding, to a local cache, the set of one or more resources;

receiving an indication of a video associated with the one or more AR effects;
causing one or more frames of the video to be rendered, with the one or more AR effects,
wherein the rendering with the one or more AR effects uses the pre-loaded set of one or more resources;

determining that at least one particular resource has been freed;
determining, based on a resource schedule, that the freed resource should be unloaded; and
in response, unloading the freed resource.

18. The computing s

Assignees

• Meta Platforms Tech Llc

Inventors

• Yoon Kyle Sejin
• Goncalves Alex Garcia
• Lim Jonathan
• Lu Yuanshuo

Classifications

• G06T1/60

  Memory management · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G11B27/031

  Electronic editing of digitised analogue information signals, e.g. audio or video signals · CPC title