Node-based image colorization on image/video editing applications

Patent metadata

Field	Value
Publication number	US-11410347-B2
Application number	US-202117196030-A
Country	US
Kind code	B2
Filing date	Mar 9, 2021
Priority date	Apr 13, 2020
Publication date	Aug 9, 2022
Grant date	Aug 9, 2022

Abstract

A system and method for node-based image colorization is provided. The system receives a first user input via a Graphical User Interface (GUI) of an image/video editing application and controls a display device to display a node graph on the GUI based on the first user input. The node graph includes an input node to select grayscale images and a colorization node which may represent a workflow for colorization of at least a first object in the grayscale images. The system receives a second user input associated with a setting of the colorization node and selects an image colorization model based on the second user input. The system executes the workflow associated with the colorization node to transmit the grayscale images to a computing device that hosts the selected image colorization model, and to receive colorized images including at least the first object colorized based on a color effect.

First claim

What is claimed is:

1. A system, comprising:
circuitry configured to:
control a display device to display a Graphical User Interface (GUI) of an image/video editing application;
receive a first user input via the Graphical User Interface (GUI);
control the display device to display a node graph on the GUI based on the first user input, wherein the node graph comprises:
an input node to select one or more grayscale images, and
a colorization node which represents a workflow for colorization of at least a first object in the one or more grayscale images;

receive, via the GUI, a second user input associated with a first setting of the colorization node;
select an image colorization model based on the received second user input; and
execute the workflow associated with the colorization node to:
transmit the one or more grayscale images to a computing device which hosts the selected image colorization model, and
receive, from the computing device, one or more colorized images comprising at least the first object colorized based on a color effect.

2. The system according to claim 1 , wherein the colorization node corresponds to a software plugin which includes program instructions for execution of the workflow.

3. The system according to claim 1 , wherein the selected image colorization model is a neural network-based colorization model, which is capable of colorizing the one or more grayscale images without a reference color image as input to the selected image colorization model.

4. The system according to claim 3 , wherein the computing device:
receives the one or more grayscale images; and
applies the selected image colorization model on the one or more grayscale images to generate the one or more colorized images.

5. The system according to claim 1 , wherein the selected image colorization model is a neural network-based colorization model, which requires a reference color image as input to colorize the one or more grayscale images.

6. The system according to claim 5 , wherein the circuitry is further configured to:
receive a third user input associated with a second setting of the colorization node;
select a reference color image comprising a region of interest based on the third user input;
execute the workflow associated with the colorization node to further transmit the selected reference color image to the computing device, wherein the computing device:
receives the selected reference color image and the one or more grayscale images; and
applies the selected image colorization model to the one or more grayscale images to transfer the color effect of the region of interest to at least the first object in the one or more grayscale images.

7. The system according to claim 1 , wherein the circuitry is further configured to control the display device to display the one or more colorized images onto the GUI of the image/video editing application.

8. A method, comprising:
controlling a display device to display a Graphical User Interface (GUI) of an image/video editing application;
receiving a first user input via the Graphical User Interface (GUI);
controlling the display device to display a node graph on the GUI based on the first user input, wherein the node graph comprises:
an input node to select one or more grayscale images, and
a colorization node which represents a workflow for colorization of at least a first object in the one or more grayscale images;

receiving, via the GUI, a second user input associated with a first setting of the colorization node;
selecting an image colorization model based on the received second user input; and
executing the workflow associated with the colorization node, wherein the execution comprises:
transmitting the one or more grayscale images to a computing device which hosts the selected image colorization model, and
receiving, from the computing device, one or more colorized images comprising at least the first object colorized based on a color effect.

9. The method according to claim 8 , wherein the colorization node corresponds to a software plugin which includes program instructions for execution of the workflow.

10. The method according to claim 8 , wherein the selected image colorization model is a neural network-based colorization model, which is capable of colorizing the one or more grayscale images without a reference color image as input to the selected image colorization model.

11. The method according to claim 10 , wherein the computing device comprising:
receiving the one or more grayscale images; and
applying the selected image colorization model on the one or more grayscale images to generate the one or more colorized images.

12. The method according to claim 8 , wherein the selected image colorization model is a neural network-based colorization model, which requires a reference color image as input to colorize the one or more grayscale images.

13. The method according to claim 12 , further comprising:
receiving a third user input associated with a second setting of the colorization node;
selecting a reference color image comprising a region of interest based on the third user input;
executing the workflow associated with the colorization node to further transmit the selected reference color image to the computing device;
receiving, by the computing device, the selected reference color image and the one or more grayscale images; and
applying, by the computing device, the selected image colorization model to the one or more grayscale images to transfer the color effect of the region of interest to at least the first object in the one or more grayscale images.

14. The method according to claim 8 , further comprising controlling the display device to display the one or more colorized images onto the GUI of the image/video editing application.

15. A non-transitory computer-readable medium having stored thereon, computer-executable instructions which, when executed a system, causes the system to execute operations, the operations comprising:
controlling a display device to display a Graphical User Interface (GUI) of an image/video editing application;
receiving a first user input via the Graphical User Interface (GUI);
controlling the display device to display a node graph on the GUI based on the first user input, wherein the node graph comprises:
an input node to select one or more grayscale images, and
a colorization node which represents a workflow for colorization of at least a first object in the one or more grayscale images;

receiving, via the GUI, a second user input associated with a first setting of the colorization node;
selecting an image colorization model based on the received second user input; and
executing the workflow associated with the colorization node, wherein the execution comprises:
transmitting the one or more grayscale images to a computing device which hosts the selected image colorization model, and
receiving, from the computing device, one or more colorized images comprising at least the first object colorized based on a color effect.

16. The non-transitory computer-readable medium according to claim 15 , wherein the colorization node corresponds to a software plugin which includes program instructions for execution of the workflow.

17. The non-transitory computer-readable medium according to claim 15 , wherein the selected image colorization model is a neural network-based colorization model, which is capable of colorizing the one or more grayscale images without a reference color image as input to the selected image colorizatio

Assignees

• Sony Group Corp

Inventors

• Lee Jong Hwa
• Tafolla Moises
• Garg Praggya
• Langley Allison
• Lyons Gary
• Stoneham Alexander
• Kim Seunghan
• Vasconcelos Luis

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06F3/0484

  for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06T11/001 Primary

  Physics · mapped topic