Image processor and method

Patent metadata

Field	Value
Publication number	US-11625815-B2
Application number	US-202017030038-A
Country	US
Kind code	B2
Filing date	Sep 23, 2020
Priority date	Dec 21, 2018
Publication date	Apr 11, 2023
Grant date	Apr 11, 2023

Abstract

An image processing apparatus and a method are provided. The apparatus comprises a plurality of processing modules configured to operate in series to refine a raw image captured by a camera, the modules comprising a first module and a second module, each of which independently implements a respective trained artificial intelligence model, wherein: the first module implements an image transformation operation that performs an operation from the set comprising: (i) an essentially pixel-level operation that increases sharpness of an image input to the module, (ii) an essentially pixel-level operation that decreases sharpness of an image input to the module, (iii) an essentially pixel-block-level operation on an image input to the module; and the second module as a whole implements a different operation from the said set.

First claim

What is claimed is:

1. An image processor comprising:
a plurality of microprocessors configured to operate in series to refine a raw image captured by a camera, the plurality of microprocessors comprising a first microprocessor, a second microprocessor, and a third microprocessor, wherein the first microprocessor implements a first convolutional neural network (CNN), the second microprocessor implements a second CNN and the third microprocessor implements a third CNN, wherein the first, second, and third CNNs are independently trained, wherein:
the first microprocessor is configured to implement a first image transformation operation that either demosaics or denoises a first image input to the first microprocessor using the first CNN;
the second microprocessor is configured to implement a second image transformation operation that demosaics or denoises a second image input to the second microprocessor using the second CNN;
the third microprocessor is configured to perform image equalizing on a third image input to the third microprocessor using the third CNN; and
each of the plurality of microprocessors, except for the first microprocessor, is configured to take an input from a preceding microprocessor so that the second image input to the second microprocessor has previously been demosaiced or denoised by the first image transformation operation implemented by the first microprocessor on the first image input, and the third image input to the third microprocessor has previously been demosaiced and denoised by the first image transformation operation implemented by the first microprocessor on the first image input and by the second image transformation operation implemented by the second microprocessor on the second image input.

2. The image processor as claimed in claim 1 , wherein the plurality of microprocessors is configured to perform an automatic white balancing operation prior to demosaicing an image.

3. The image processor as claimed in claim 1 , wherein the plurality of microprocessors is configured to perform demosaicing of the image after de-noising the image.

4. The image processor as claimed in claim 1 , wherein each of the first and second image transformation operations is an essentially pixel-level operation.

5. The image processor as claimed in claim 1 , wherein the image equalizing performed by the third microprocessor is an essentially pixel-block-level operation that comprises colour mapping.

6. The image processor as claimed in claim 1 , wherein the image processor is configured to generate an output that is a compressed representation of the first image input.

7. The image processor as claimed in claim 1 , wherein the plurality of microprocessors further comprises a fourth microprocessor configured to perform multi-frame noise reduction.

8. The image processor as claimed in claim 1 , wherein the plurality of microprocessors further comprises a fourth microprocessor configured to increase the contrast to noise ratio of short exposure images.

9. The image processor as claimed in claim 1 , wherein the plurality of microprocessors further comprises a fourth microprocessor configured to enhance detail in the image.

10. The image processor as claimed in claim 1 , wherein the plurality of microprocessors further comprises a fourth microprocessor configured to perform super-resolution of the image.

11. The image processor as claimed in claim 1 , wherein the plurality of microprocessors further comprises a fourth microprocessor configured to perform an image sharpening operation.

12. The image processor as claimed in claim 1 , wherein the first or second CNN used by the respective microprocessor to demosaic the respective input image is trained to demosaic the respective input image based on the content thereof.

13. A method for refining a raw image captured by a camera, wherein the method is applied to an image processor comprising a plurality of microprocessors configured to operate in series, wherein the plurality of microprocessors comprises a first microprocessor, a second microprocessor, and a third microprocessor, and wherein the first microprocessor implements a first convolutional neural network (CNN), the second microprocessor implements a second CNN, and the third microprocessor implements a third CNN, wherein the first, second and third CNNs are independently trained, the method comprising:
implementing, by the first microprocessor, a first image transformation operation that either demosaics or denoises a first image input to the first microprocessor using the first CNN;
implementing, by the second microprocessor, a second image transformation operation that demosaics or denoises a second image input to the second microprocessor using the second CNN; and
performing, by the third microprocessor, image equalizing on a third image input to the third microprocessor using the third CNN; and
wherein each of the plurality of microprocessors, except for the first microprocessor, is configured to take an input from a preceding microprocessor so that the second image input to the second microprocessor has previously been demosaiced or denoised by the first image transformation operation implemented by the first microprocessor on the first image input, and the third image input to the third microprocessor has previously been demosaiced and denoised by the first image transformation operation implemented by the first microprocessor on the first image input and by the second image transformation operation implemented by the second microprocessor on the second image input.

14. The method as claimed in claim 13 , wherein the plurality of microprocessors operating in series further perform an automatic white balancing operation prior to demosaicing an image.

15. The method as claimed in claim 13 , wherein the plurality of microprocessors operating in series further perform demosaicing of the image after de-noising the image.

16. The method as claimed in claim 13 , wherein each of the first and second image transformation operations is an essentially pixel-level operation.

17. The method as claimed in claim 13 , wherein the image equalizing performed by the third microprocessor is an essentially pixel-block-level operation that comprises colour mapping.

18. The method as claimed in claim 13 , further comprising:
generating an output that is a compressed representation of the first image input.

19. A device comprising:
a chip integrated with a plurality of neural processors thereon;
the plurality of neural processors are configured to operate in series to refine a raw image captured by a camera,
wherein the plurality of neural processors comprises a first neural processor, a second neural processor, and a third neural processor,
wherein each of the first neural processor implements a first convolutional neural network (CNN), the second neural processor implements a second CNN, and the third neural processor implements a third CNN, and the first, second and third CNNs are independently trained, and

the first neural processor is configured to implement a first image transformation operation that either demosaics or denoises a first image input to the first neural processor using the first CNN;
the second neural processor is configured to implement a second image transformation operation that demosaics or denoises a second image input to the second neural processor using the second CNN;
the third neural processor is configured to perform image equalizing on a third image input to the third neural processor using the third CN

Assignees

• Huawei Tech Co Ltd

Inventors

• Slabaugh Gregory
• Yan Youliang
• Song Fenglong
• Chen Gang
• Li Jiangwei
• Wang Tao
• Liu Liu
• Alexiou Ioannis
• Marras Ioannis
• Moran Sean
• Mcdonagh Steven George
• Pereira Jose Costa
• Smirnov Viktor Vladimirovich

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• H04N23/88

  for colour balance, e.g. white-balance circuits or colour temperature control · CPC title

• H04N9/646

  for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters · CPC title

• G06T3/4015

  Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns · CPC title