Tile-selection based deep demosaicing acceleration

Patent metadata

Field	Value
Publication number	US-10863148-B2
Application number	US-201816010313-A
Country	US
Kind code	B2
Filing date	Jun 15, 2018
Priority date	Jun 15, 2018
Publication date	Dec 8, 2020
Grant date	Dec 8, 2020

Abstract

Aspects of the present disclosure relate to systems and methods for selectively processing tiles of a captured image. The captured image may be parsed into a plurality of tiles. A subset of the plurality of tiles may be selected based at least in part on at least one metric. The selected tiles may be processed according to a first technique, while each unselected tile may be selectively processed according to one or more techniques different from the first technique. The first technique and the one or more techniques may each be either an image demosaicing technique or an image denoising technique.

First claim

What is claimed is:

1. A method for selectively processing tiles of a captured image, the method comprising:
parsing pixels of the captured image into a plurality of tiles where each tile only includes a set of contiguously positioned pixels of the captured image, each tile defining a respective spatial portion of the captured image;
selecting a subset of tiles among the plurality of tiles based at least in part on at least one metric;
processing the selected subset of tiles according to a first technique; and
selectively processing each unselected tile among the plurality of tiles according to one or more techniques that is different from the first technique,
wherein the first technique and the one or more techniques are each one from the group consisting of an image demosaicing technique and an image denoising technique.

2. The method of claim 1 , further comprising capturing, by a color filter array (CFA) image sensor, an image to obtain the captured image wherein the captured image is a mosaiced image.

3. The method of claim 2 , wherein the at least one metric comprises at least one from the group consisting of a sum of absolute differences (SAD) metric and a pixel saturation (SAT) metric.

4. The method of claim 3 , wherein selecting the subset of tiles of the plurality of tiles comprises selecting each tile having the SAD metric greater than a SAD threshold.

5. The method of claim 3 , wherein selecting the subset of tiles of the plurality of tiles comprises selecting each tile having the SAT metric less than a SAT threshold.

6. The method of claim 2 , wherein processing the selected subset of tiles according to the first technique comprises demosaicing the selected subset of tiles using a deep neural network.

7. The method of claim 6 , wherein selectively processing each unselected tile comprises processing each unselected tile using a hardware-based demosaicing technique.

8. The method of claim 1 , further comprising generating a combined output image based on a combination of the selected subset of tiles and the unselected tiles using a blending operation.

9. The method of claim 8 , further comprising assigning weights to respective portions of adjacent tiles that overlap one another and blending the respective portions based on the assigned weights.

10. The method of claim 1 , wherein the subset of tiles of the plurality of tiles is selected based at least in part on a sum of absolute differences between each tile and a corresponding tile of a temporally previous captured image.

11. An image processing device configured to selectively process tiles of a captured image, comprising:
a memory configured to store image data; and
one or more processors implemented in circuitry and in communication with the memory, the one or more processors being configured to:
parse pixels of the captured image into a plurality of tiles where each tile only includes a set of contiguously positioned pixels of the captured image, each tile defining a respective spatial portion of the captured image;
select a subset of tiles among the plurality of tiles based at least in part on at least one metric;
process the selected subset of tiles according to a first technique; and
selectively process each unselected tile among the plurality of tiles according to one or more techniques that is different from the first technique,
wherein the first technique and the one or more techniques are each one from the group consisting of an image demosaicing technique and an image denoising technique.

12. The image processing device of claim 11 , further comprising a color filter array (CFA) image sensor configured to capture an image to obtain the captured image wherein the captured image is a mosaiced image.

13. The image processing device of claim 12 , wherein the at least one metric comprises at least one from the group consisting of a sum of absolute differences (SAD) metric and a pixel saturation (SAT) metric.

14. The image processing device of claim 13 , wherein the one or more processors is further configured to select tiles having the SAD metric greater than a SAD threshold as the subset of tiles.

15. The image processing device of claim 13 , wherein the one or more processors is further configured to select tiles having the SAT metric less than a SAT threshold as the subset of tiles.

16. The image processing device of claim 12 , wherein the one or more processors is further configured to demosaic the selected subset of tiles using a deep neural network.

17. The image processing device of claim 16 , wherein the one or more processors is configured to process each unselected tile using a hardware-based demosaicing technique.

18. The image processing device of claim 11 , wherein the one more processors is further configured to generate a combined output image based on a combination of the selected subset of tiles and the unselected tiles using a blending operation.

19. The image processing device of claim 18 , wherein the one or more processors is further configured to assign weights to respective portions of adjacent tiles that overlap one another and to blend the respective portions based on the assigned weights.

20. The image processing device of claim 11 , wherein the subset of tiles of the plurality of tiles is selected based at least in part on a sum of absolute differences between each tile and a corresponding tile of a temporally previous captured image.

21. A non-transitory, computer-readable storage medium storing instructions that, when executed by one or more processors of an image processing device, cause the image processing device to:
parse pixels of a captured image into a plurality of tiles where each tile only includes a set of contiguously positioned pixels of the captured image, each tile defining a respective spatial portion of the captured image;
select a subset of tiles among the plurality of tiles based at least in part on at least one metric;
process the selected subset of tiles according to a first technique; and
selectively process each unselected tile among the plurality of tiles according to one or more techniques that is different from the first technique,
wherein the first technique and the one or more techniques are each one from the group consisting of an image demosaicing technique and an image denoising technique.

22. The non-transitory, computer-readable storage medium of claim 21 , wherein the at least one metric comprises one from the group consisting of a sum of absolute differences (SAD) metric and a pixel saturation (SAT) metric.

23. The non-transitory, computer-readable storage medium of claim 22 , further comprising instructions that, when executed, cause the one or more processors to select tiles having the SAD metric greater than a SAD threshold.

24. The non-transitory, computer-readable storage medium of claim 22 , further comprising instructions that, when executed, cause the one or more processors to select tiles having the SAT metric greater than a SAT threshold.

25. The non-transitory, computer-readable storage medium of claim 21 , further comprising instructions that, when executed, cause the one or more processors to demosaic the selected subset of tiles using a deep neural network.

26. The non-transitory, computer-readable storage medium of claim 25 , further comprising instructions that, when executed, cause the one or more processors to process each unselected

Assignees

• Qualcomm Inc

Inventors

• Chuang Shang-Chih
• Jiang Xiaoyun
• Liu jun zuo

Classifications

• G06T3/4015 Primary

  Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns · CPC title

• H04N23/10

  for generating image signals from different wavelengths · CPC title

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title