Switchable propagation neural network

Patent metadata

Field	Value
Publication number	US-11328169-B2
Application number	US-201916353835-A
Country	US
Kind code	B2
Filing date	Mar 14, 2019
Priority date	Sep 26, 2017
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

A temporal propagation network (TPN) system learns the affinity matrix for video image processing tasks. An affinity matrix is a generic matrix that defines the similarity of two points in space. The TPN system includes a guidance neural network model and a temporal propagation module and is trained for a particular computer vision task to propagate visual properties from a key-frame represented by dense data (color), to another frame that is represented by coarse data (grey-scale). The guidance neural network model generates an affinity matrix referred to as a global transformation matrix from task-specific data for the key-frame and the other frame. The temporal propagation module applies the global transformation matrix to the key-frame property data to produce propagated property data (color) for the other frame. For example, the TPN system may be used to colorize several frames of greyscale video using a single manually colorized key-frame.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receiving property data for the pixels in the key-frame;
receiving task-specific data for a frame of the video sequence defining attributes of pixels in the frame;
processing, according to parameters, by a guidance neural network model, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises at least two affinity values that are aligned in either the horizontal or vertical pixel dimension of the key-frame for transitions from the key-frame to the frame; and
applying the guidance data to the property data for the pixels in the key-frame to generate property data for the pixels in the frame, wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.

2. The computer-implemented method of claim 1 , wherein the guidance data further comprises task-specific affinity values for transitions from the key-frame to other frames in the video sequence.

3. The computer-implemented method of claim 1 , wherein the guidance neural network model is trained during generation of the property data for the frame using a training dataset for the task.

4. The computer-implemented method of claim 1 , wherein the task is colorization, the attributes are lightness data, the property data for the key-frame is color data corresponding to the key-frame and the property data for the frame comprises color data corresponding to the frame.

5. The computer-implemented method of claim 1 , wherein the task is segmentation, the attributes are color data, the property data for the key-frame is segmentation data corresponding to the key-frame and the property data for the frame comprises segmentation data corresponding to the frame.

6. A computer-implemented method, comprising:
receiving task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receiving task-specific data for a frame of the video sequence defining attributes of pixels in the frame;
processing, according to parameters, by a guidance neural network model, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises a global transformation matrix that is regularized as orthogonal; and
applying the guidance data to property data for the key-frame to generate property data for the frame.

7. A system, comprising a processor configured to:
implement a guidance neural network model that is configured to:
receive task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receive task-specific data for a frame of the video sequence defining attributes of pixels in the frame; and
process, according to parameters, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises a global transformation matrix that is regularized as orthogonal; and
apply the guidance data to property data for the key-frame to generate property data for the frame.

8. A computer-implemented method, comprising:
receiving task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receiving task-specific data for a frame of the video sequence defining attributes of pixels in the frame;
processing, according to parameters, by a guidance neural network model, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task; and
applying the guidance data to property data for the key-frame to generate property data for the frame, wherein the task is conversion to high dynamic range, the attributes are low dynamic range data, the property data for the key-frame is high dynamic range data corresponding to the key-frame and the property data for the frame comprises high dynamic range data corresponding to the frame.

9. The computer-implemented method of claim 8 , wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.

10. A system, comprising:
a processor configured to:
implement a guidance neural network model that is configured to:
receive task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receive property data for the pixels in the key-frame;
receive task-specific data for a frame of the video sequence defining attributes of pixels in the frame; and
process, according to parameters, the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises at least two affinity values that are aligned in either the horizontal or vertical pixel dimension of the key-frame for transitions from the key-frame to the frame; and
apply the guidance data to the property data for the pixels in the key-frame to generate property data for the pixels in the frame, wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.

11. The computer-implemented method of claim 6 , wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.

12. The system of claim 10 , wherein the guidance data further comprises task-specific affinity values for transitions from the key-frame to other frames in the video sequence.

13. The system of claim 10 , wherein the guidance neural network model is trained during generation of the property data for the frame using a training dataset for the task.

14. The system of claim 10 , wherein the task is colorization, the attributes are lightness data, the property data for the key-frame is color data corresponding to the key-frame and the property data for the frame comprises color data corresponding to the frame.

15. The system of claim 10 , wherein the task is segmentation, the attributes are color data, the property data for the key-frame is segmentation data corresponding to the key-frame and the property data for the frame comprises segmentation data corresponding to the frame.

16. A non-transitory computer-readable media storing computer instructions for spatial linear propagation that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving task-specific data for a key-frame of a video sequence defining attributes of pixels in the key-frame;
receiving property data for the pixels in the key-frame;
receiving task-specific data for a frame of the video sequence defining attributes of pixels in the frame;
processing, according to parameters the task-specific data for a key-frame and the task-specific data for the frame to produce guidance data for a task, wherein the guidance data comprises at least two affinity values that are aligned in either the horizontal or vertical pixel dimension of the key-frame for transitions from the key-frame to the frame; and
applying the guidance data to the property data for the pixels in the key-frame to generate property data for the pixels in the frame, wherein the guidance data preserves a style energy of the property data for the key-frame in the generated property data for the frame.

17. The non-

Assignees

• Nvidia Corp

Inventors

• Liu Sifei
• De Mello Shalini
• Gu Jinwei
• Jampani Varun
• Kautz Jan

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/56

  relating to colour · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title