Method and apparatus of neural network based image signal processor

Patent metadata

Field	Value
Publication number	US-10460231-B2
Application number	US-201615075076-A
Country	US
Kind code	B2
Filing date	Mar 18, 2016
Priority date	Dec 29, 2015
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

An image signal processing (ISP) system is provided. The system includes a neural network trained by inputting a set of raw data images and a correlating set of desired quality output images; the neural network including an input for receiving input image data and providing processed output; wherein the processed output includes input image data that has been adjusted for at least one image quality attribute. A method and an imaging device are disclosed.

First claim

What is claimed is:

1. An image signal processing (ISP) system, comprising:
a neural network that receives an input image and outputs a processed image, the input image including input image data, the neural network comprising a convolutional neural network having multiple parallel paths in which each path scales the input image data to separate the input image data into separate frequency bands of spatial resolution, a first path comprising a first convolutional layer having an output at a first frequency band of spatial resolution, the output of the first convolutional layer being pooled at a second frequency band of spatial resolution that is less than the first frequency band of spatial resolution, a second path being coupled to the pooled output of the first convolutional layer and comprising a second convolutional layer having an output at the second frequency band of spatial resolution, the output of the second convolutional layer being pooled at a third frequency band of spatial resolution that is less than the second frequency band of spatial resolution, a third path being coupled to the pooled output of the second convolutional layer and comprising a third convolutional layer having an output at the third frequency band of spatial resolution, an output of the third convolutional layer being upsampled to the second frequency band of spatial resolution and concatenated with the output of the second convolutional layer to form a first intermediate output, the first intermediate output being upsampled to the first frequency band of spatial resolution and concatenated with the output of the first convolutional layer to form a second intermediate output, the second intermediate output being input to a fourth convolutional layer to output the processed image, and the neural network adjusting the input image data for at least one image-quality attribute to generate the processed image; and
an image-processing chain that receives the processed image output from the neural network, the image-processing chain providing at least one of color correction, gamma correction, edge enhancement and contrast enhancement of the processed image to output a final image,
wherein the at least one adjusted image-quality attribute comprises an image size, an aspect ratio, a brightness, an intensity, a bit depth, a white value, a dynamic range, a gray level, a contouring, a smoothing, a speckle, a color space values, an interleaving, a gamma correction, a contrast enhancement, a sharpness and a demosaicing.

2. The image signal processing (ISP) system as in claim 1 , wherein the neural network has been trained by a back-propagation technique.

3. The image signal processing (ISP) system as in claim 1 , wherein the processed image includes at least one detected feature comprising at least one of a line, a curve, an intensity and a color within the input image.

4. The image signal processing (ISP) as in claim 1 , wherein the convolutional neural network comprises three parallel paths.

5. The image signal processing (ISP) system as in claim 1 , wherein the input image includes noise,
wherein the neural network includes a plurality of layers,
wherein the neural network includes a plurality of filters each having dimensions that are different from each other,
wherein a height and a width of each respective filter is different from other filters,
wherein the neural network includes a pooling layer, and
wherein a filter is shifted across a predetermined layer of the neural network by a predetermined stride s.

6. A method for providing an image signal processing (ISP) system, the method comprising:
inputting an input image into a neural network, the input image comprising input image data, and the neural network comprising a convolutional neural network having multiple parallel paths in which each path scales the input image data to separate the input image data into multiple separate frequency bands of spatial resolution, a first path comprising a first convolutional layer having an output at a first frequency band of spatial resolution, the output of the first convolutional layer being pooled at a second frequency band of spatial resolution that is less than the first frequency band of spatial resolution, a second path being coupled to the pooled output of the first convolutional layer and comprising a second convolutional layer having an output at the second frequency band of spatial resolution, the output of the second convolutional layer being pooled at a third frequency band of spatial resolution that is less than the second frequency band of spatial resolution, a third path being coupled to the pooled output of the second convolutional layer and comprising a third convolutional layer having an output at the third frequency band of spatial resolution, an output of the third convolutional layer being upsampled to the second frequency band of spatial resolution and concatenated with the output of the second convolutional layer to form a first intermediate output, the first intermediate output being upsampled to the first frequency band of spatial resolution and concatenated with the output of the first convolutional layer to form a second intermediate output, the second intermediate output being input to a fourth convolutional layer to output the processed image;
adjusting the input image by the neural network to output a processed image having input image data adjusted for at least one image-quality attribute; and,
generating by an image-processing chain a final image from the processed image, the image-processing chain processing at least one of a color correction, a gamma correction, an edge enhancement and a contrast enhancement of the processed image,
wherein the at least one adjusted image-quality attribute comprises an image size, an aspect ratio, a brightness, an intensity, a bit depth, a white value, a dynamic range, a gray level, a contouring, a smoothing, a speckle, a color space values, an interleaving, a gamma correction, a contrast enhancement, a sharpness and a demosaicing.

7. The method as in claim 6 , further comprising training the neural network by back-propagation.

8. The method as in claim 6 , further comprising generating the input image using an imaging device.

9. The method as in claim 6 , wherein the processed image includes at least one detected feature comprising at least one of a line, a curve, an intensity and a color within the input image.

10. The method as in claim 6 , wherein the input image data includes noise,
wherein the neural network includes a plurality of layers,
wherein the neural network includes three parallel paths,
wherein the neural network includes a plurality of filters each having dimensions that are different from each other,
wherein a height and a width of each respective filter is different from other filters,
wherein the neural network includes a pooling layer, and
wherein a filter is shifted across a predetermined layer of the neural network by a predetermined stride s.

11. An imaging device, comprising:
an imaging sensor that outputs raw image data; and
an image signal processing (ISP) system comprising a convolutional neural network and an image-processing chain, the convolutional neural network comprising multiple parallel paths in which each path scales the raw image data to separate the raw image data into separate frequency bands of spatial resolution, a first path comprising a first convolutional layer having an output at a first frequency band of spatial resolution, the output of the first convolutional layer being pooled at a second frequency band of spatial resolution that is less than the first frequency band of spatial resolution, a second path be

Assignees

• Zhang Qiang
• Ji Zhengping
• Wang Yibing Michelle
• Ovsiannikov Ilia
• Samsung Electronics Co Ltd

Inventors

• Zhang Qiang
• Ji Zhengping
• Wang Yibing Michelle
• Ovsiannikov Ilia

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F18/24133

  Distances to prototypes · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title