Flexible resolution support for image and video style transfer

Patent metadata

Field	Value
Publication number	US-10909657-B1
Application number	US-201816032844-A
Country	US
Kind code	B1
Filing date	Jul 11, 2018
Priority date	Sep 11, 2017
Publication date	Feb 2, 2021
Grant date	Feb 2, 2021

Abstract

Artistic styles extracted from one or more source images may be applied to one or more target images, e.g., in the form of stylized images and/or stylized video sequences. The extracted artistic style may be stored as a plurality of layers in a neural network, which neural network may be further optimized, e.g., via the fusion of various elements of the network's architectures. An optimized network architecture may be determined for each processing environment in which the network will be applied. The artistic style may be applied to the obtained images and/or video sequence of images using various optimization methods, such as the use of scalars to control the resolution of the unstylized and stylized images, temporal consistency constraints, as well as the use of dynamically adjustable or selectable versions of Deep Neural Networks (DNN) that are responsive to system performance parameters, such as available processing resources and thermal capacity.

First claim

What is claimed is:

1. An image processing method, comprising:
obtaining a first artistic style, wherein the first artistic style is stored as a plurality of layers in a neural network, and wherein the neural network is configured to operate on images having a fixed first resolution;
obtaining a first target image captured at a first time, wherein the first target image has a second resolution, and wherein the second resolution is different than the first resolution;
scaling the obtained first target image from the second resolution to the first resolution;
applying the first artistic style to the first target image to create a stylized version of the first target image;
obtaining a second target image captured at a second time, wherein the second target image has the first resolution;
applying the first artistic style to the second target image to create a stylized version of the second target image; and
storing the stylized versions of the first and second target images in a memory.

2. The method of claim 1 , further comprising:
scaling the stylized version of the first target image to a third resolution prior to storing the stylized version of the first target image in the memory.

3. The method of claim 2 , wherein the third resolution is larger than the first and second resolutions.

4. The method of claim 1 , wherein the first artistic style is selected by a user.

5. The method of claim 1 , wherein:
scaling the obtained first target image from the second resolution to the first resolution comprises at least one of the following: adjusting a pixel aspect ratio (PAR) of the obtained first target image; using a Lanczos filter; and
using a bilateral filter.

6. The method of claim 1 , further comprising:
obtaining a third target image captured at a third time, wherein the third target image has a third resolution, and wherein the third resolution is different than both the first resolution and the second resolution;
scaling the obtained third target image from the third resolution to the first resolution;
applying the first artistic style to the third target image to create a stylized version of the third target image; and
storing the stylized version of the third target image in the memory.

7. The method of claim 1 , further comprising:
displaying the stylized version of the first target image on a display screen.

8. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
obtain a first artistic style, wherein the first artistic style is stored as a plurality of layers in a neural network, and wherein the neural network is configured to operate on images having a fixed first resolution;
obtain a first target image captured at a first time, wherein the first target image has a second resolution, and wherein the second resolution is different than the first resolution;
scale the obtained first target image from the second resolution to the first resolution;
apply the first artistic style to the first target image to create a stylized version of the first target image;
obtain a second target image captured at a second time, wherein the second target image has the first resolution;
apply the first artistic style to the second target image to create a stylized version of the second target image; and
store the stylized versions of the first and second target images in a memory.

9. The non-transitory program storage device of claim 8 , further comprising instructions stored thereon to cause one or more processors to:
scale the stylized version of the first target image to a third resolution prior to storing the stylized version of the first target image in the memory.

10. The non-transitory program storage device of claim 9 , wherein the third resolution is larger than the first resolution and the second resolution.

11. The non-transitory program storage device of claim 8 , wherein the instructions to scale the obtained first target image from the second resolution to the first resolution further comprise instructions to cause the one or more processors to perform at least one of the following operations: adjusting a pixel aspect ratio (PAR) of the obtained first target image; using a Lanczos filter; and using a bilateral filter.

12. The non-transitory program storage device of claim 8 , further comprising instructions stored thereon to cause one or more processors to:
obtain a third target image captured at a third time, wherein the third target image has a third resolution, and wherein the third resolution is different than both the first resolution and the second resolution;
scale the obtained third target image from the third resolution to the first resolution;
apply the first artistic style to the third target image to create a stylized version of the third target image; and
store the stylized version of the third target image in the memory.

13. The non-transitory program storage device of claim 8 , wherein the first artistic style is selected by a user.

14. A device, comprising:
an image sensor;
a display screen;
a memory communicatively coupled to the image sensor;
one or more processors operatively coupled to the image sensor and the memory configured to execute instructions causing the one or more processors to:
obtain an indication from a user of the device to apply a first artistic style to images captured by the image sensor, wherein the first artistic style is stored as a plurality of layers in a neural network, and wherein the neural network is configured to operate on images having a fixed first resolution;
obtain an indication from the user to capture a first target image with the image sensor at a first time, wherein the first target image has a second resolution, and wherein the second resolution is different than the first resolution;
scale the obtained first target image from the second resolution to the first resolution;
apply the first artistic style to the first target image to create a stylized version of the first target image;
obtain a second target image captured at a second time, wherein the second target image has the first resolution;
apply the first artistic style to the second target image to create a stylized version of the second target image; and
display the stylized versions of the first and second target images on the display screen.

15. The device of claim 14 , wherein the one or more processors are further configured to execute instructions causing the one or more processors to:
scale the stylized version of the first target image to a third resolution prior to displaying the stylized version of the first target image on the display screen.

16. The device of claim 15 , wherein the third resolution is larger than the first and second resolutions.

17. The device of claim 15 , wherein the first artistic style is selected by a user.

18. The device of claim 14 , wherein the instructions to scale the obtained first target image from the second resolution to the first resolution further comprise instructions to cause the one or more processors to perform at least one of the following operations: adjusting a pixel aspect ratio (PAR) of the obtained first target image; using a Lanczos filter; and using a bilateral filter.

19. The device of claim 14 , wherein the one or more processors are further configured to execute instructions causing the one or more processors to:
obtain a third target image captured at a third time, wherein the third target image has a third resolution, and wherein the third

Assignees

• Apple Inc

Inventors

• Rossi Francesco
• Wang Xiaohuan C
• Walsh Brian E
• Rymkowski Bartlomiej W
• Shi Xiaojin
• Zuliani Marco
• Marinichev Alexey
• Poulain Benjamin
• Khalili Omid

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title