Using portrait images in augmented reality components

Patent metadata

Field	Value
Publication number	US-11869164-B2
Application number	US-202217804008-A
Country	US
Kind code	B2
Filing date	May 25, 2022
Priority date	Nov 11, 2020
Publication date	Jan 9, 2024
Grant date	Jan 9, 2024

Abstract

The technical problem of creating an augmented reality (AR) experience that, on one hand, is accessible from a camera view user interface provided with a messaging client and that, also, can perform a modification based on a previously captured image of a user, is addressed by providing an AR component. When a user, while accessing the messaging client, engages a user selectable element representing the AR component in the camera view user interface, the messaging system loads the AR component in the messaging client. The AR component comprises a target media content object, which can be animation or live action video. The loaded AR component accesses a portrait image associated with a user and modifies the target media content using the portrait image. The resulting target media content object is displayed in the camera view user interface.

First claim

What is claimed is:

1. A method comprising:
causing presentation of an initial camera view user interface at a client device, the initial camera view user interface including an output of a digital image sensor of a camera of the client device, a user selectable element actionable to perform image capture by the camera and a user selectable element actionable to apply a visual effect produced by an augmented reality component;
detecting activation, from the initial camera view user interface, of the user selectable element actionable to apply the visual effect produced by the augmented reality component; in response to the detecting:
accessing a face object derived from a previously obtained portrait image,
generating a modified target media content object by replacing a portion of an initial target media content object with the face object, and
generating a modified camera view user interface by overlaying the modified target media content object in the initial camera view user interface over at least one portion of the output of the digital image sensor of the camera; and
causing presentation of the modified camera view user interface at the client device.

2. The method of claim 1 , wherein the initial target media content object includes animation.

3. The method of claim 1 , wherein the initial target media content object includes live action video.

4. The method of claim 1 , comprising configuring the augmented reality component in a messaging system for exchanging data over a network.

5. The method of claim 4 , wherein the previously obtained portrait image is associated with a user profile representing a user in the messaging system.

6. The method of claim 5 , wherein the augmented reality component includes one or more machine learning models for face animation configured to re-render the face object derived from the previously obtained portrait image of the user profile with one or more modifications, wherein the generating of the modified target media content object comprises executing the one or more machine learning models.

7. The method of claim 5 , wherein the user profile represents the user accessing the initial camera view user interface at the client device.

8. The method of claim 5 , wherein the user profile represents a connection of the user accessing the initial camera view user interface at the client device.

9. The method of claim 1 , comprising determining spatial placement of the modified target media content object in the modified camera view user interface based on movement of a user detected by the digital image sensor of the camera.

10. The method of claim 1 , wherein the modified camera view user interface comprises multiple instances of the modified target media content object.

11. A system comprising:
one or more processors; and
a non-transitory computer readable storage medium comprising instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:
causing presentation of an initial camera view user interface at a client device, the initial camera view user interface including an output of a digital image sensor of a camera of the client device, a user selectable element actionable to perform image capture by the camera and a user selectable element actionable to apply a visual effect produced by an augmented reality component;
detecting activation, from the initial camera view user interface, of the user selectable element actionable to apply the visual effect produced by the augmented reality component; in response to the detecting:
accessing a face object derived from a previously obtained portrait image,
generating a modified target media content object by replacing a portion of an initial target media content object with the face object, and
generating a modified camera view user interface by overlaying the modified target media content object in the initial camera view user interface over at least one portion of the output of the digital image sensor of the camera; and
causing presentation of the modified camera view user interface at the client device.

12. The system of claim 11 , wherein the initial target media content object includes animation.

13. The system of claim 11 , wherein the initial target media content object includes live action video.

14. The system of claim 11 , wherein the operations caused by the instructions executed by the one or more processors include configuring the augmented reality component in a messaging system for exchanging data over a network.

15. The system of claim 14 , wherein the previously obtained portrait image is associated with a user profile representing a user in the messaging system.

16. The system of claim 15 , wherein the augmented reality component includes one or more machine learning models for face animation configured to re-render the face object derived from the previously obtained portrait image of the user profile with one or more modifications, wherein the generating of the modified target media content object comprises executing the one or more machine learning models.

17. The system of claim 15 , wherein the user profile represents the user accessing the initial camera view user interface at the client device.

18. The system of claim 15 , wherein the user profile represents a connection of the user accessing the initial camera view user interface at the client device.

19. The system of claim 11 , wherein the operations caused by the instructions executed by the one or more processors include determining spatial placement of the modified target media content object in the modified camera view user interface based on movement of a user detected by the digital image sensor of the camera.

20. A machine-readable non-transitory storage medium having instruction data executable by a machine to cause the machine to perform operations comprising:
causing presentation of an initial camera view user interface at a client device, the initial camera view user interface including an output of a digital image sensor of a camera of the client device, a user selectable element actionable to perform image capture by the camera and a user selectable element actionable to apply a visual effect produced by an augmented reality component;
detecting activation, from the initial camera view user interface, of the user selectable element actionable to apply the visual effect produced by the augmented reality component; in response to the detecting:
accessing a face object derived from a previously obtained portrait image,
generating a modified target media content object by replacing a portion of an initial target media content object with the face object, and
generating a modified camera view user interface by overlaying the modified target media content object in the initial camera view user interface over at least one portion of the output of the digital image sensor of the camera; and
causing presentation of the modified camera view user interface at the client device.

Assignees

• Snap Inc

Inventors

• Golobokov Roman
• Mashrabov Aleksandr
• Matov Dmitry
• Voss Jeremy Baker

Classifications

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title