Video processing with preview of AR effects

Patent metadata

Field	Value
Publication number	US-12361658-B2
Application number	US-202217946643-A
Country	US
Kind code	B2
Filing date	Sep 16, 2022
Priority date	Jul 28, 2022
Publication date	Jul 15, 2025
Grant date	Jul 15, 2025

Abstract

Image augmentation effects are provided on a device that includes a display and a camera. A simplified augmented reality effect is applied to a stream of images captured by the camera, to generate a preview stream of images. The preview stream of images is displayed on the display. A second stream of images corresponding to the first stream of images is saved to an initial video file. A full augmented reality effect, corresponding to the simplified augmented reality affect, is then applied to the second stream of images to generate a fully-augmented stream of images, which are saved to a further video file. The further video file can then be played back on the display to show the final, fully augmented reality effect as applied to the stream of images.

First claim

The invention claimed is:

1. A method, executed by one or more processors, for providing image augmentation effects on a device including a display and at least one camera, the method comprising:
receiving a first stream of images captured by the at least one camera;
applying a simplified augmented reality effect to the first stream of images captured by the at least one camera, to generate a preview stream of images;
displaying the preview stream of images on the display;
saving a second stream of images corresponding to the first stream of images captured by the at least one camera to an initial video file, wherein the second stream of images is a parallel stream of images to the first stream of images;
retrieving the second stream of images from the initial video file;
applying a full augmented reality effect corresponding to the simplified augmented reality effect to the second stream of images to generate a fully-augmented stream of images; and
saving the fully-augmented stream of images to a further video file.

2. The method of claim 1 , wherein the retrieving of the second stream is initiated in response to receipt of user input requesting playback of an enhanced version of the first stream of images.

3. The method of claim 1 , wherein the retrieving of the second stream of images from the initial video file begins automatically on completion of the saving of the initial video file, the method further comprising:
playing back the further video file on the display automatically once the further video file has been saved.

4. The method of claim 3 , wherein the full augmented reality effect is based on a machine learning model.

5. The method of claim 1 , wherein the full augmented reality effect is based on a machine learning model.

6. The method of claim 1 , wherein the second stream of images is of a higher resolution than the first stream of images.

7. The method of claim 1 , wherein the second stream of images comprises a video-encoded version of the first stream of images.

8. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations for providing image augmentation effects on a device including a display and at least one camera, the operations comprising:
receiving a first stream of images captured by the at least one camera;
applying a simplified augmented reality effect to the first stream of images captured by the at least one camera, to generate a preview stream of images;
displaying the preview stream of images on the display;
saving a second stream of images corresponding to the first stream of images captured by the at least one camera to an initial video file, wherein the second stream of images is a parallel stream of images to the first stream of images;
retrieving the second stream of images from the initial video file;
applying a full augmented reality effect corresponding to the simplified augmented reality effect to the second stream of images to generate a fully-augmented stream of images; and
saving the fully-augmented stream of images to a further video file.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the retrieving of the second stream is initiated in response to receipt of user input requesting playback of an enhanced version of the first stream of images.

10. The non-transitory computer-readable storage medium of claim 8 , wherein the retrieving of the second stream of images from the initial video file begins automatically on completion of the saving of the initial video file, the operations further comprising:
playing back the further video file on the display automatically once the further video file has been saved.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the full augmented reality effect is based on a machine learning model.

12. The non-transitory computer-readable storage medium of claim 8 , wherein the full augmented reality effect is based on a machine learning model.

13. The non-transitory computer-readable storage medium of claim 8 , wherein the second stream of images is of a higher resolution than the first stream of images.

14. The non-transitory computer-readable storage medium of claim 8 , wherein the second stream of images comprises a video-encoded version of the first stream of images.

15. A computing device comprising:
at least one camera;
a display;
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, configure the device to perform operations for providing image augmentation effects, the operations comprising:
receiving a first stream of images captured by the at least one camera;
applying a simplified augmented reality effect to the first stream of images captured by the at least one camera, to generate a preview stream of images;
displaying the preview stream of images on the display; and
saving a second stream of images corresponding to the first stream of images captured by the at least one camera to an initial video file, wherein the second stream of images is a parallel stream of images to the first stream of images;
retrieving the second stream of images from the initial video file;
applying a full augmented reality effect corresponding to the simplified augmented reality effect to the second stream of images to generate a fully-augmented stream of images; and
saving the fully-augmented stream of images to a further video file.

16. The computing device of claim 15 , wherein the retrieving of the second stream is initiated in response to receipt of user input requesting playback of an enhanced version of the first stream of images.

17. The computing device of claim 15 , wherein the retrieving of the second stream of images from the initial video file begins automatically on completion of the saving of the initial video file, the operations further comprising:
playing back the further video file on the display automatically once the further video file has been saved.

18. The computing device of claim 17 , wherein the full augmented reality effect is based on a machine learning model.

19. The computing device of claim 15 , wherein the full augmented reality effect is based on a machine learning model.

20. The computing device of claim 15 , wherein the second stream of images is of a higher resolution than the first stream of images.

Assignees

• Snap Inc

Inventors

• Zhu Cai
• Liu Chuangwen
• Wu Haoyun
• Yuan Weihao

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06N20/00

  Machine learning · CPC title

• G11B27/031

  Electronic editing of digitised analogue information signals, e.g. audio or video signals · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• H04N21/432

  Content retrieval operation from a local storage medium, e.g. hard-disk {(details of retrieval of video data and associated meta data in video databases G06F16/739)} · CPC title