Video color propagation

Patent metadata

Field	Value
Publication number	US-10623709-B2
Application number	US-201816119792-A
Country	US
Kind code	B2
Filing date	Aug 31, 2018
Priority date	Aug 31, 2018
Publication date	Apr 14, 2020
Grant date	Apr 14, 2020

Abstract

A video processing system includes a computing platform having a hardware processor and a memory storing a software code including a convolutional neural network (CNN). The hardware processor executes the software code to receive video data including a key video frame in color and a video sequence in gray scale, determine a first estimated colorization for each frame of the video sequence except the key video frame based on a colorization of a previous frame, and determine a second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color. For each frame of the video sequence except the key video frame, the software code further blends the first estimated colorization with the second estimated colorization using a color fusion stage of the CNN to produce a colorized video sequence corresponding to the video sequence in gray scale.

First claim

What is claimed is:

1. A video processing system comprising:
a computing platform including a hardware processor and a system memory storing a software code including a convolutional neural network (CNN);
the hardware processor configured to execute the software code to:
receive a video data including a key video frame in color and a video sequence in gray scale beginning with the key video frame;
determine a first estimated colorization for each frame of the video sequence except the key video frame based on a colorization of a previous neighboring frame of the video sequence;
determine a second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color; and
for each frame of the video sequence except the key video frame, blend the first estimated colorization for the each frame with the second estimated colorization for the each frame using a color fusion stage of the CNN to produce a colorized video sequence corresponding to the video sequence in gray scale.

2. The video processing system of claim 1 , wherein the hardware processor is further configured to execute the software code to render the colorized video sequence on a display.

3. The video processing system of claim 1 , wherein the first estimated colorization for each frame of the video sequence except the key video frame is determined based on the colorization of the previous neighboring frame of the video sequence using a local color propagation stage of the CNN.

4. The video processing system of claim 3 , wherein the local color propagation stage of the CNN is trained using a warp loss function.

5. The video processing system of claim 3 , wherein the second estimated colorization for each frame of the video sequence except the key video frame is determined based on the key video frame in color using a global color transfer stage of the CNN.

6. The video processing system of claim 5 , wherein the second estimated colorization for each frame of the video sequence except the key video frame is determined using the global color transfer stage of the CNN further based on a comparison of a feature map of the each frame with a feature map of the key video frame in gray scale.

7. The video processing system of claim 5 , wherein the first estimated colorization for each frame of the video sequence except the key video frame and the second estimated colorization for each frame of the video sequence except the key video frame are determined in parallel by the respective local color propagation stage and the global color transfer stage of the CNN.

8. The video processing system of claim 1 , wherein the CNN is trained using a loss function including an image loss term and a warp loss term.

9. A method for use by a video processing system including a computing platform having a hardware processor and a system memory storing a software code including a convolutional neural network (CNN), the method comprising:
receiving, using the hardware processor, a video data including a key video frame in color and a video sequence in gray scale beginning with the key video frame;
determining, using the hardware processor, a first estimated colorization for each frame of the video sequence except the key video frame based on a colorization of a previous neighboring frame of the video sequence;
determining, using the hardware processor, a second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color; and
for each frame of the video sequence except the key video frame, blend the first estimated colorization for the each frame with the second estimated colorization for the each frame, using the hardware processor and a color fusion stage of the CNN, to produce a colorized video sequence corresponding to the video sequence in gray scale.

10. The method of claim 9 , further comprising rendering, using the hardware processor, the colorized video sequence on a display.

11. The method of claim 9 , wherein determining the first estimated colorization for each frame of the video sequence except the key video frame based on the colorization of the previous neighboring frame of the video sequence is performed using a local color propagation stage of the CNN.

12. The method of claim 11 , wherein the local color propagation stage of the CNN is trained using a warp loss function.

13. The method of claim 11 , wherein determining the second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color is performed using a global color transfer stage of the CNN.

14. The method of claim 13 , wherein determining the second estimated colorization for each frame of the video sequence except the key video frame using the global color transfer stage of the CNN is further based on a comparison of a feature map of the each frame with a feature map of the key video frame in gray scale.

15. The method of claim 13 , wherein determining the first estimated colorization for each frame of the video sequence except the key video frame and determining the second estimated colorization for each frame of the video sequence except the key video frame are performed in parallel by the respective local color propagation stage and the global color transfer stage of the CNN.

16. The method of claim 9 , wherein the CNN is trained using a loss function including an image loss term and a warp loss term.

17. A video compression method for use by a computing platform having a hardware processor, the method comprising:
receiving, using the hardware processor, a video data including a key video frame in color and a video sequence in gray scale beginning with the key video frame;
determining, using the hardware processor, a first estimated colorization for each frame of the video sequence except the key video frame based on a colorization of a previous neighboring frame of the video sequence;
determining, using the hardware processor, a second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color; and
for each frame of the video sequence except the key video frame, blend the first estimated colorization for the each frame with the second estimated colorization for the each frame, using the hardware processor and a color fusion stage of a convolutional neural network (CNN), to produce a colorized video sequence corresponding to the video sequence in gray scale.

18. The video compression method of claim 17 , wherein determining the first estimated colorization for each frame of the video sequence except the key video frame based on the colorization of the previous neighboring frame of the video sequence is performed using a local color propagation stage of the CNN.

19. The video compression method of claim 18 , wherein the local color propagation stage of the CNN is trained using a warp loss function.

20. The video compression method of claim 18 , wherein determining the second estimated colorization for each frame of the video sequence except the key video frame based on the key video frame in color is performed using a global color transfer stage of the CNN.

Assignees

• Disney Entpr Inc
• Eth Zuerich

Inventors

• Schroers Christopher
• Meyer Simone
• Cornillere Victor
• Gross Markus
• Djelouah Abdelaziz

Classifications

• G06T7/90 Primary

  Determination of colour characteristics · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10024

  Color image · CPC title

• H04N9/43 Primary

  Conversion of monochrome picture signals to colour picture signals for colour picture display · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title