Machine learning for visual processing

Patent metadata

Field	Value
Publication number	US-11528492-B2
Application number	US-201715679984-A
Country	US
Kind code	B2
Filing date	Aug 17, 2017
Priority date	Feb 19, 2015
Publication date	Dec 13, 2022
Grant date	Dec 13, 2022

Abstract

A method for developing an enhancement model for low-quality visual data, the method comprising the steps of receiving one or more sections of higher-quality visual data; and training a hierarchical algorithm. The hierarchical algorithm is operable to increase the quality of one or more sections of lower-quality visual data so as to substantially reproduce the one or more sections of higher-quality visual data. The hierarchical algorithm is then outputted.

First claim

What is claimed is:

1. A method for reducing amount of data to be transferred when communicating visual data over a network from a first node to a second node, the method comprising:
reducing a resolution of a section of higher-resolution visual data to provide a corresponding section of lower-resolution visual data;
developing a neural network model to increase the resolution of the corresponding section of lower-resolution visual data, the developing including training the neural network model to increase the resolution of the lower-resolution visual data to substantially reproduce the higher-resolution visual data, wherein the training includes processing sections of lower-resolution visual data using the neural network model to produce an output from the model, calculating an error by comparing sections of higher-resolution visual data corresponding to the sections of lower-resolution visual data with the output obtained with the neural network model, the error being quantified by a pre-defined cost function, and adjusting parameters associated with the neural network model to minimize the error;
transmitting the corresponding section of lower-resolution visual data to the second node; and
transmitting to the second node the parameters associated with the developed neural network model that is trained using the sections of lower-resolution of visual data,
wherein the transmitted section of lower-resolution visual data and the parameters associated with the developed neural network model thereby enable the second node to substantially reproduce the section of higher-resolution visual data from the transmitted section of lower-resolution visual data visual data using the developed neural network model that is trained using the sections of lower-resolution of visual data.

2. The method of claim 1 , wherein the developed neural network model includes or a convolutional neural network model.

3. The method of claim 1 , further comprising converting the visual data into a sequence of images before the resolution is reduced.

4. The method of claim 1 , wherein the section of higher-resolution visual data includes one of: a single frame, a sequence of frames, or a region within a frame or sequence of frames.

5. The method of claim 1 , further comprising dividing the section of higher-resolution visual data into smaller sections based on similarities between a plurality of frames of the section of higher-resolution visual data.

6. A method for increasing a resolution of a section of lower-resolution visual data communicated over a network from a first node to a second node, the method comprising:
receiving the section of lower-resolution visual data via the network;
receiving a corresponding developed neural network model operable to increase the resolution of the lower-resolution visual data, the developed neural network model having been developed by training the neural network model to increase the resolution of the lower-resolution visual data to substantially reproduce higher-resolution visual data, wherein the training includes processing sections of lower-resolution visual data using the neural network model calculating an error by comparing sections of higher-resolution visual data corresponding to the sections of lower-resolution visual data with output obtained with the neural network model, the error being quantified by a pre-defined cost function, and adjusting parameters associated with the neural network model to minimize the error; and
using the developed neural network model to increase the resolution of the section of lower-resolution visual data to substantially recreate a section of higher-resolution visual data.

7. The method of claim 6 , wherein the section of lower-resolution visual data has a greater amount of artefacts than the section of higher-resolution visual data.

8. The method of claim 6 , wherein increasing the resolution of the section of lower-resolution visual data includes upscaling the resolution of the section of lower-resolution visual data.

9. The method of claim 6 , wherein the section of lower-resolution visual data includes at least one of: an image, a sequence of images, or a section of video.

10. The method of claim 6 , wherein the section of lower-resolution visual data includes at least one of: a single frame of visual data, a sequence of frames of visual data, or a region within a frame or sequence of frames of visual data.

11. A system for reducing an amount of data transferred when communicating visual data over a network, the system comprising two or more nodes, wherein a first node is configured to:
reduce a resolution of a section of higher-resolution visual data to provide a corresponding section of lower-resolution visual data;
develop a neural network model to increase the resolution of the corresponding section of lower-resolution visual data, the developing including training the neural network model to increase the resolution of the lower-resolution visual data to substantially reproduce the higher-resolution visual data, wherein the training includes processing sections of lower-resolution visual data using the developed neural network model to produce an output from the model, calculating an error by, comparing sections of higher-resolution visual data corresponding to the sections of lower-resolution visual data and the output obtained with the neural network model, the error being quantified by a pre-defined cost function, and adjusting parameters associated with the neural network model to minimize the error;
transmit the corresponding section of lower-resolution visual data to a second node; and
transmit to the second node the parameters associated with the developed neural network model that is trained using the sections of lower-resolution of visual data,
wherein the second node is configured to:
receive the corresponding section of lower-resolution visual data via the network;
receive the developed neural network model; and
use the developed neural network model to increase the resolution of the section of lower-resolution visual data to substantially recreate the section of higher-resolution visual data.

12. The system of claim 11 , wherein transmitting the section of lower-resolution visual data to the second node and transmitting to the second node the developed neural network model occur substantially simultaneously.

13. The system of claim 11 , wherein the developed neural network model is developed for the section of lower-resolution visual data that is transferred over the network.

14. The system of claim 11 , wherein the developed neural network model includes a plurality of connected layers.

15. The system of claim 14 , wherein the layers are sequential.

16. The system of claim 15 , wherein a last layer of the layers is operable to increase the resolution of the section of lower-resolution visual data.

17. The system of claim 11 , wherein the developed neural network model uses a spatio-temporal approach.

18. The system of claim 11 , further comprising using down-sampling to reduce a resolution of the higher-resolution visual data to that of the section of the lower-resolution visual data.

Assignees

• Magic Pony Tech Limited
• Twitter Inc

Inventors

• Wang Zehan
• Bishop Robert David
• Shi Wenzhe
• Caballero Jose
• Aitken Andrew Peter
• Totz Johannes

Classifications

• H04N19/36 Primary

  Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability · CPC title

• G06T3/4046 Primary

  using neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• H04N19/59

  involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution · CPC title