Objective-effectuators in synthesized reality settings

Patent metadata

Field	Value
Publication number	US-12380334-B2
Application number	US-202017070332-A
Country	US
Kind code	B2
Filing date	Oct 14, 2020
Priority date	Jan 22, 2018
Publication date	Aug 5, 2025
Grant date	Aug 5, 2025

Abstract

Various implementations disclosed herein include devices, systems, and methods for presenting objective-effectuators in synthesized reality settings. In various implementations, a device includes a non-transitory memory and one or more processors coupled with the non-transitory memory. In some implementations, a method includes instantiating an objective-effectuator into a synthesized reality setting. In some implementations, the objective-effectuator is characterized by a set of predefined actions and a set of visual rendering attributes. In some implementations, the method includes obtaining an objective for the objective-effectuator. In some implementations, the method includes determining contextual information characterizing the synthesized reality setting. In some implementations, the method includes generating a sequence of actions from the set of predefined actions based on the contextual information and the objective. In some implementations, the method includes modifying the objective-effectuator based on the sequence of actions.

First claim

What is claimed is:

1. A method comprising:
at a device including a non-transitory memory, a display, and one or more processors coupled with the non-transitory memory:
while presenting a synthesized reality setting on the display:
instantiating an objective-effectuator into the synthesized reality setting, wherein the objective-effectuator is characterized by a set of predefined actions and a set of visual rendering attributes, and the objective-efffectuator represents a character from a source material;
obtaining an objective for the objective-effectuator;
determining contextual information characterizing the synthesized reality setting at least in part by determining a mapping between the synthesized reality setting and a physical setting in which the device is located;
generating a sequence of actions from the set of predefined actions based on the contextual information and the objective, wherein the actions in the sequence of actions are within a degree of similarity to actions that the character performs in the source material; and
manipulating the objective-effectuator to perform the sequence of actions.

2. The method of claim 1 , wherein generating the sequence of actions comprises utilizing a neural network to generate the sequence of actions.

3. The method of claim 2 , wherein the neural network generates the sequence of actions based on a set of neural network parameters.

4. The method of claim 3 , further comprising:
adjusting the set of neural network parameters based on the sequence of actions.

5. The method of claim 3 , further comprising:
determining the set of neural network parameters based on a reward function that assigns positive rewards to desirable actions and negative rewards to undesirable actions.

6. The method of claim 2 , further comprising:
configuring the neural network based on reinforcement learning.

7. The method of claim 2 , further comprising:
training the neural network based on one or more of videos, novels, books, comics, and video games associated with the objective-effectuator.

8. The method of claim 1 , wherein manipulating the objective-effectuator comprises:
providing the sequence of actions to a display pipeline in order to output synthesized reality content showing the objective-effectuator performing the sequence of actions within the synthesized reality setting.

9. The method of claim 1 , further comprising:
obtaining the set of predefined actions from the source material including one or more of movies, video games, comics, and novels.

10. The method of claim 9 , wherein obtaining the set of predefined actions comprises scraping the source material to extract the set of predefined actions; and
wherein generating the sequence of actions comprises selecting the actions in the sequence from the set of predefined actions.

11. The method of claim 9 , wherein obtaining the set of predefined actions comprises:
determining the set of predefined actions based on a type of the objective-effectuator that is instantiated.

12. The method of claim 9 , wherein obtaining the set of predefined actions comprises:
determining the set of predefined actions based on a user-specified configuration of the objective-effectuator.

13. The method of claim 9 , wherein obtaining the set of predefined actions comprises:
determining the set of predefined actions based on limits specified by an entity that owns the object.

14. The method of claim 1 , further comprising:
capturing an image; and
obtaining the set of visual rendering attributes from the image.

15. The method of claim 1 , wherein obtaining the objective comprises:
receiving a user input that indicates the objective.

16. The method of claim 1 , wherein obtaining the objective comprises:
receiving the objective from a content engine that generates plots for the object.

17. The method of claim 1 , wherein the contextual information indicates whether other objective-effectuators have been instantiated within the synthesized reality setting.

18. The method of claim 1 , wherein generating the sequence of actions comprises:
generating a first action in response to the contextual information indicating that a second objective-effectuator has been instantiated within the synthesized reality setting; and
generating a second action that is different from the first action in response to the contextual information indicating that a third objective-effectuator has been instantiated within the synthesized reality setting.

19. A device comprising:
one or more processors;
a non-transitory memory;
one or more displays; and
one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to:
while presenting a synthesized reality setting on the one or more displays:
instantiate an objective-effectuator into the synthesized reality setting, wherein the objective-effectuator is characterized by a set of predefined actions and a set of visual rendering attributes, and the objective-efffectuator represents a character from a source material;
obtain an objective for the objective-effectuator;
determine contextual information characterizing the synthesized reality setting at least in part by determining a mapping between the synthesized reality setting and a physical setting in which the device is located;
generate a sequence of actions from the set of predefined actions based on the contextual information and the objective, wherein the actions in the sequence of actions are within a degree of similarity to actions that the character performs in the source material; and
manipulate the objective-effectuator to perform the sequence of actions.

20. The device of claim 19 , wherein the one or more programs, which, when executed by the one or more processors, further cause the device to:
obtain the set of predefined actions from the source material including one or more of movies, video games, comics, and novels.

21. The device of claim 19 , wherein obtaining the objective comprises:
receiving the objective from a content engine that generates plots for the object.

22. The device of claim 19 , wherein the contextual information indicates whether other objective-effectuators have been instantiated within the synthesized reality setting.

23. The device of claim 19 , wherein generating the sequence of actions comprises:
generating a first action in response to the contextual information indicating that a second objective-effectuator has been instantiated within the synthesized reality setting; and
generating a second action that is different from the first action in response to the contextual information indicating that a third objective-effectuator has been instantiated within the synthesized reality setting.

24. A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with a display, cause the device to:
while presenting a synthesized reality setting on the display:
instantiate an objective-effectuator into the synthesized reality setting, wherein the objective-effectuator is characterized by a set of predefined actions and a set of visual rendering attributes, and the objective-efffectuator represents a character from a source material;
obtain an objective for the objective-effectuator;
determine contextual information characterizing the synthesized reality setting at least in part by determining a

Assignees

• Apple Inc

Inventors

• Richter Ian M
• Saini Amritpal Singh
• Soares Olivier

Classifications

• G06N3/088 Primary

  Non-supervised learning, e.g. competitive learning · CPC title

• G06T15/005 Primary

  General purpose rendering architectures · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title