Real-time tracking-compensated image effects

Patent metadata

Field	Value
Publication number	US-10474900-B2
Application number	US-201715706096-A
Country	US
Kind code	B2
Filing date	Sep 15, 2017
Priority date	Sep 15, 2017
Publication date	Nov 12, 2019
Grant date	Nov 12, 2019

Abstract

A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.

First claim

What is claimed is:

1. A method comprising:
generating, using an image sensor, an image sequence;
generating modified keyframe images by processing alternating images from the image sequence using a machine learning scheme, the alternating images from the image sequence being keyframe images that are separated by one or more non-keyframe images;
generating modified non-keyframe images using flow maps generated between the non-keyframe images and preceding keyframe images, the modified non-keyframe images being generated by applying the flow maps to the modified keyframe images;
displaying, on a display device, a modified image sequence generated using the modified keyframe images and the modified non-keyframe images.

2. The method of claim 1 , wherein the modified keyframe images are generated in a first pipeline and the modified non-keyframe images are generated in a second pipeline, wherein the second pipeline is asynchronous from the first pipeline.

3. The method of claim 2 , wherein the machine learning scheme is a convolutional neural network.

4. The method of claim 1 , further comprising:
publishing the modified image sequence as an ephemeral message on a social media network.

5. The method of claim 1 , wherein each flow map is generated by a non-keyframe image and a preceding keyframe image, the non-keyframe image and the preceding keyframe image separated by one or more additional non-keyframe images.

6. The method of claim 1 , wherein the preceding keyframe images used to generate the flow maps have existing modified keyframe images, and the flow maps are applied to the existing modified keyframe images.

7. The method of claim 1 , wherein the modified image sequence is displayed on the display device in one or more of: real time or near real time.

8. The method of claim 1 , wherein the modified keyframe images and the modified non-keyframe images are ordered in the modified image sequence according to an ordering of images in the image sequence.

9. The method of claim 1 , further comprising:
generating the modified image sequence by using the modified keyframe images and the modified non-keyframe images.

10. The method of claim 9 , wherein the modified keyframe images and the modified non-keyframe images are image masks used to apply one or more visual effects to the image sequence.

11. The method of claim 1 , further comprising:
generating the flow maps using a motion tracking scheme that describes transformations of image features across a plurality of images.

12. A system comprising:
one or more processors of a machine;
a display device; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
generating an image sequence;
generating modified keyframe images by processing alternating images from the image sequence using a machine learning scheme, the alternating images from the image sequence being keyframe images that are separated by one or more non-keyframe images;
generating modified non-keyframe images using flow maps generated between the non-keyframe images and preceding keyframe images, the modified non-keyframe images being generated by applying the flow maps to the modified keyframe images; and
displaying, on the display device, a modified image sequence generated using the modified keyframe images and the modified non-keyframe images.

13. The system of claim 12 , wherein the modified keyframe images are generated in a first pipeline and the modified non-keyframe images are generated in a second pipeline, wherein the second pipeline is asynchronous from the first pipeline.

14. The system of claim 13 , wherein the machine learning scheme is a convolutional neural network.

15. The system of claim 13 , wherein the first pipeline and the second pipeline operate in parallel.

16. The system of claim 12 , wherein each flow map is generated by a non-keyframe image and a preceding keyframe image, the non-keyframe image and the preceding keyframe image separated by one or more additional non-keyframe images.

17. The system of claim 12 , wherein the preceding keyframe images used to generate the flow maps have existing modified keyframe images, and the flow maps are applied to the existing modified keyframe images.

18. The system of claim 12 , wherein the modified image sequence is displayed on the machine in one or more of: real time or near real time.

19. A non-transitory machine readable medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
generating an image sequence;
generating modified keyframe images by processing alternating images from the image sequence using a machine learning scheme, the alternating images from the image sequence being keyframe images that are separated by one or more non-keyframe images;
generating modified non-keyframe images using flow maps generated between the non-keyframe images and preceding keyframe images, the modified non-keyframe images being generated by applying the flow maps to the modified keyframe images; and
displaying, on a display device, a modified image sequence generated using the modified keyframe images and the modified non-keyframe images.

20. The non-transitory machine readable medium of claim 19 , wherein the modified keyframe images are generated in a first pipeline and the modified non-keyframe images are generated in a second pipeline, wherein the second pipeline is asynchronous to the first pipeline.

Assignees

• Snap Inc

Inventors

• Hare Samuel Edward
• Poliakov Fedir
• Wang Guohui
• Xiong Xuehan
• Yang Jianchao
• Yang Linjie
• Anilkumar Shah Tanmay

Classifications

• G06T7/248

  involving reference images or patches · CPC title

• G06T2200/28

  involving image processing hardware · CPC title

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20081

  Training; Learning · CPC title