Full reference image quality assessment based on convolutional neural network

Patent metadata

Field	Value
Publication number	US-9741107-B2
Application number	US-201514732518-A
Country	US
Kind code	B2
Filing date	Jun 5, 2015
Priority date	Jun 5, 2015
Publication date	Aug 22, 2017
Grant date	Aug 22, 2017

Abstract

Embodiments generally relate to providing systems and methods for assessing image quality of a distorted image relative to a reference image. In one embodiment, the system comprises a convolutional neural network that accepts as an input the distorted image and the reference image, and provides as an output a metric of image quality. In another embodiment, the method comprises inputting the distorted image and the reference image to a convolutional neural network configured to process the distorted image and the reference image and provide as an output a metric of image quality.

First claim

We claim:

1. A system for assessing image quality of a distorted image relative to a reference image, the system comprising:
a convolutional neural network that accepts as an input the distorted image and the reference image, and provides as an output a metric of image quality;
wherein the convolutional neural network comprises a plurality of layers comprising:
an input layer configured to apply a normalizing function to image patches making up each of the distorted image and the reference image to provide a normalized distorted image and a normalized reference image; and
a convolution layer configured to convolve each of the normalized distorted image and the normalized reference image with N1 filters to provide N1 pairs of feature maps, each pair containing one filtered normalized distorted image and one correspondingly filtered and normalized reference image, where N1 is an integer greater than unity.

2. The system of claim 1 wherein the plurality of layers further comprises:
a linear combination layer configured to compute N2 linear combinations of the N1 feature maps provided from each of the normalized distorted image and the normalized reference image, providing N2 pairs of combined feature maps, each pair containing one combination of filtered normalized distorted images and one corresponding combination of filtered and normalized reference images, where N2 is an integer greater than unity;
a similarity computation layer configured to compute N2 similarity maps, each similarity map based on corresponding pixels from a different one of the N2 pairs of combined feature maps; and
a pooling layer configured to apply an average pooling for each of the N2 similarity maps to provide N2 similarity input values.

3. The system of claim 2 wherein the plurality of layers further comprises:
a fully connected layer configured to act on the N2 similarity input values to provide M hidden node values, where M is an integer greater than N2; and
a linear regression layer configured to map the M hidden node values to a single output node to provide the metric of image quality.

4. The system of claim 3 wherein N1=N2 and N1=10.

5. The system of claim 3 wherein M=800.

6. The system of claim 1 wherein a squared activation function is applied at each node of the N1 pairs of feature maps before the N1 pairs of feature maps are provided by the convolution layer to any subsequent layer of the plurality of layers.

7. A method for assessing image quality of a distorted image relative to a reference image, the method comprising:
inputting the distorted image and the reference image to a convolutional neural network configured to process the distorted image and the reference image and provide as an output a metric of image quality;
wherein the processing performed by the convolutional neural network comprises:
applying a normalizing function to image patches making up each of the distorted image and the reference image to provide a normalized distorted image and a normalized reference image; and
convolving each of the normalized distorted image and the normalized reference image with N1 filters to provide N1 pairs of feature maps, each pair containing one filtered normalized distorted image and one correspondingly filtered and normalized reference image, where N1 is an integer greater than unity.

8. The method of claim 7 further comprising:
computing N2 linear combinations of the N1 feature maps provided from each of the normalized distorted image and the normalized reference image, providing N2 pairs of combined feature maps, each pair containing one combination of filtered normalized distorted images and one corresponding combination of filtered and normalized reference images;
computing N2 similarity maps, each similarity map based on corresponding pixels from a different one of the N2 pairs of combined feature maps; and
applying an average pooling for each of the N2 similarity maps to provide N2 similarity input values.

9. The method of claim 8 further comprising:
inputting the N2 similarity input values to a fully connected layer to provide M hidden node values, where M is an integer greater than N2; and
mapping the M hidden node values through a regression layer to provide the metric of image quality.

10. The method of claim 9 where M=800.

Assignees

• Xu Xun
• Ye Peng
• Sony Corp

Inventors

• Xu Xun
• Ye Peng

Classifications

• G06T7/001 Primary

  using an image reference approach · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T7/0002 Primary

  Inspection of images, e.g. flaw detection · CPC title

• G06F18/24137

  Distances to cluster centroïds · CPC title