Real-time tracking-compensated image effects

Patent metadata

Field	Value
Publication number	US-11989938-B2
Application number	US-202318312479-A
Country	US
Kind code	B2
Filing date	May 4, 2023
Priority date	Sep 15, 2017
Publication date	May 21, 2024
Grant date	May 21, 2024

Abstract

A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.

First claim

What is claimed is:

1. A method comprising:
generating, using one or more processors of a device, a video sequence comprising a plurality of frames;
detecting a lag of an editing engine configured to apply a machine learning scheme to the video sequence, a previous frame having a corresponding modified previous frame and a current frame that does not have a corresponding modified current frame;
in response to detecting the lag, generating a map between the previous frame and the current frame; and
generating the corresponding modified current frame by applying the map to the modified previous frame.

2. The method of claim 1 , wherein detecting the lag of the editing engine comprises:
determining that the editing engine is still processing images preceding the current frame, where the current frame is a next image to be processed.

3. The method of claim 1 , further comprising:
generating modified images in a first pipeline by applying the machine learning scheme to the video sequence, the first pipeline comprising the previous frame having the corresponding modified previous frame and the current frame that does not have the corresponding modified current frame in the first pipeline.

4. The method of claim 3 , further comprising:
in response to detecting the lag, generating, in a second pipeline, the map between the previous frame and the current frame.

5. The method of claim 4 , wherein the second pipeline is asynchronous to the first pipeline.

6. The method of claim 4 , wherein the first pipeline and the second pipeline are implemented on different threads of the one or more processors of the device.

7. The method of claim 1 , wherein the previous frame and the current frame are separated by a plurality of other frames in the video sequence.

8. The method of claim 1 , wherein the machine learning scheme is trained to apply an image manipulation, and wherein the modified previous frame exhibits the image manipulation.

9. The method of claim 1 , wherein the map is a flow map that describes changes of image features in the video sequence.

10. The method of claim 1 , further comprising:
displaying a modified video sequence on the device, the modified video sequence comprising the modified previous frame and the modified current frame,
wherein the modified video sequence collates the modified previous frame and the modified current frame.

11. A device comprising:
one or more processors;
a memory storing instructions that, when executed by the one or more processors, cause the device to perform operations comprising:
generating, using one or more processors of a device, a video sequence comprising a plurality of frames;
detecting a lag of an editing engine configured to apply a machine learning scheme to the video sequence, a previous frame having a corresponding modified previous frame and a current frame that does not have a corresponding modified current frame;
in response to detecting the lag, generating a map between the previous frame and the current frame; and
generating the corresponding modified current frame by applying the map to the modified previous frame.

12. The device of claim 11 , wherein detecting the lag of the editing engine comprises:
determining that the editing engine is still processing images preceding the current frame, where the current frame is a next image to be processed.

13. The device of claim 11 , wherein the operations further comprise:
generating modified images in a first pipeline by applying the machine learning scheme to the video sequence, the first pipeline comprising the previous frame having the corresponding modified previous frame and the current frame that does not have the corresponding modified current frame in the first pipeline.

14. The device of claim 13 , wherein the operations further comprise:
in response to detecting the lag, generating, in a second pipeline, the map between the previous frame and the current frame.

15. The device of claim 14 , wherein the second pipeline is asynchronous to the first pipeline.

16. The device of claim 14 , wherein the first pipeline and the second pipeline are implemented on different threads of the one or more processors of the device.

17. The device of claim 11 , wherein the previous frame and the current frame are separated by a plurality of other frames in the video sequence.

18. The device of claim 11 , wherein the machine learning scheme is trained to apply an image manipulation, and wherein the modified previous frame exhibits the image manipulation.

19. The device of claim 11 , wherein the map is a flow map that describes changes of image features in the video sequence.

20. A non-transitory machine readable medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
generating, using one or more processors of a device, a video sequence comprising a plurality of frames;
detecting a lag of an editing engine configured to apply a machine learning scheme to the video sequence, a previous frame having a corresponding modified previous frame and a current frame that does not have a corresponding modified current frame;
in response to detecting the lag, generating a map between the previous frame and the current frame; and
generating the corresponding modified current frame by applying the map to the modified previous frame.

Assignees

• Snap Inc

Inventors

• Hare Samuel Edward
• Poliakov Fedir
• Wang Guohui
• Xiong Xuehan
• Yang Jianchao
• Yang Linjie
• Anilkumar Shah Tanmay

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T7/215

  Motion-based segmentation · CPC title