Using portrait images in augmented reality components

Patent metadata

Field	Value
Publication number	US-11354872-B2
Application number	US-202016949700-A
Country	US
Kind code	B2
Filing date	Nov 11, 2020
Priority date	Nov 11, 2020
Publication date	Jun 7, 2022
Grant date	Jun 7, 2022

Abstract

The technical problem of creating an augmented reality (AR) experience that, on one hand, is accessible from a camera view user interface provided with a messaging client and that, also, can perform a modification based on a previously captured image of a user, is addressed by providing an AR component. When a user, while accessing the messaging client, engages a user selectable element representing the AR component in the camera view user interface, the messaging system loads the AR component in the messaging client. The AR component comprises a target media content object, which can be animation or live action video. The loaded AR component accesses a portrait image associated with a user and modifies the target media content using the portrait image. The resulting target media content object is displayed in the camera view user interface.

First claim

What is claimed is:

1. A method comprising:
in a messaging system for exchanging data over a network, configuring an augmented reality component to modify, in real time, presentation of an output of a digital image sensor of a camera displayed in an initial camera view user interface by overlaying a modified target media content object over at least one portion of the output of the digital image sensor of the camera;
causing presentation of the initial camera view user interface at a client device, the initial camera view user interface including the output of the digital image sensor of the camera of the client device, a user selectable element actionable to perform image capture by the camera and a user selectable element actionable to selectively apply or remove a visual effect produced by the augmented reality component;
detecting activation, from the initial camera view user interface, of the user selectable element actionable to selectively apply or remove the visual effect produced by the augmented reality component;
in response to the detecting of activation, from the initial camera view user interface, of the user selectable element actionable to selectively apply or remove the visual effect produced by the augmented reality component:
accessing a face object derived from a previously obtained portrait image associated with a user profile representing a user in the messaging system, and
generating the modified target media content object by replacing a portion of an initial target media content object with the face object,

generating a modified camera view user interface, the modified camera view user interface comprising the modified target media content object overlaid over the at least one portion of the output of the digital image sensor of the camera and the user selectable element actionable to perform image capture by the camera and the user selectable element actionable to selectively apply or remove the visual effect produced by the augmented reality component; and
causing presentation of the modified camera view user interface at the client device.

2. The method of claim 1 , wherein the initial target media content object includes animation.

3. The method of claim 1 , wherein the initial target media content object includes live action video.

4. The method of claim 1 , comprising determining spatial placement of the modified target media content object in the modified camera view user interface based on movement of the user detected by the digital image sensor of the camera.

5. The method of claim 1 , wherein the augmented reality component includes one or more machine learning models for face animation configured to re-render the face object derived from the previously obtained portrait image of the user profile with one or more modifications, wherein the generating of the modified target media content object comprises executing the one or more machine learning models.

6. The method of claim 1 , wherein the modified camera view user interface comprises multiple instances of the modified target media content object.

7. The method of claim 1 , wherein the user profile represents the user accessing the initial camera view user interface at the client device.

8. The method of claim 1 , wherein the user profile represents a connection of the user accessing the initial camera view user interface at the client device.

9. A system comprising:
one or more processors; and
a non-transitory computer readable storage medium comprising instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:
in a messaging system for exchanging data over a network, configuring an augmented reality component to modify, in real time, presentation of an output of a digital image sensor of a camera displayed in an initial camera view user interface by overlaying a modified target media content object over at least one portion of the output of the digital image sensor of the camera;
causing presentation of the initial camera view user interface at a client device, the initial camera view user interface including the output of the digital image sensor of the camera of the client device, a user selectable element actionable to perform image capture by the camera and a user selectable element actionable to selectively apply or remove a visual effect produced by the augmented reality component;
detecting activation, from the initial camera view user interface, of the user selectable element actionable to selectively apply or remove the visual effect produced by the augmented reality component;
in response to the detecting of activation, from the initial camera view user interface, of the user selectable element actionable to selectively apply or remove the visual effect produced by the augmented reality component:
accessing a face object derived from a previously obtained portrait image associated with a user profile representing a user in the messaging system, and
generating the modified target media content object by replacing a portion of an initial target media content object with the face object,

generating a modified camera view user interface, the modified camera view user interface comprising the modified target media content object overlaid over the at least one portion of the output of the digital image sensor of the camera and the user selectable element actionable to perform image capture by the camera and the user selectable element actionable to selectively apply or remove the visual effect produced by the augmented reality component; and
causing presentation of the modified camera view user interface at the client device.

10. The system of claim 9 , wherein the initial target media content object includes animation.

11. The system of claim 9 , wherein the initial target media content object includes live action video.

12. The system of claim 9 , wherein the operations caused by the instructions executed by the one or more processors further include determining spatial placement of the modified target media content object in the modified camera view user interface based on movement of the user detected by the digital image sensor of the camera.

13. The system of claim 9 , wherein the augmented reality component includes one or more machine learning models for face animation configured to re-render the face object derived from the previously obtained portrait image of the user profile with one or more modifications, wherein the generating of the modified target media content object comprises executing the one or more machine learning models.

14. The system of claim 9 , wherein the modified camera view user interface comprises multiple instances of the modified target media content object.

15. The system of claim 9 , wherein the user profile represents the user accessing the initial camera view user interface at the client device.

16. A machine-readable non-transitory storage medium having instruction data executable by a machine to cause the machine to perform operations comprising:
in a messaging system for exchanging data over a network, configuring an augmented reality component to modify, in real time, presentation of an output of a digital image sensor of a camera displayed in an initial camera view user interface by overlaying a modified target media content object over at least one portion of the output of the digital image sensor of the camera;
causing presentation of the initial camera view user interface at a client device, the initial camera view user interface including the output of the digital image sensor o

Assignees

• Snap Inc

Inventors

• Golobokov Roman
• Mashrabov Aleksandr
• Matov Dmitry
• Voss Jeremy Baker

Classifications

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• H04L51/04

  Real-time or near real-time messaging, e.g. instant messaging [IM] · CPC title

• G06T2219/2024

  Style variation · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title