Low-light imaging using trained convolutional neural networks

Patent metadata

Field	Value
Publication number	US-10803565-B2
Application number	US-201816031152-A
Country	US
Kind code	B2
Filing date	Jul 10, 2018
Priority date	Jul 10, 2018
Publication date	Oct 13, 2020
Grant date	Oct 13, 2020

Abstract

An example apparatus for imaging in low-light environments includes a raw sensor data receiver to receive raw sensor data from an imaging sensor. The apparatus also includes a convolutional neural network trained to generate an illuminated image based on the received raw sensor data. The convolutional neural network is trained based on images captured by a sensor similar to the imaging sensor.

First claim

What is claimed is:

1. An apparatus for imaging in low-light environments, comprising:
a raw sensor data receiver to receive raw sensor data from an imaging sensor;
a black level subtractor to subtract a black level from packed data to generate black-level-subtracted packed data, the black level generated from the raw sensor data;
an amplifier to amplify the black-level-subtracted packed data based on a received amplification ratio to generate amplified black-level-subtracted packed data to be input into a convolutional neural network; and
the convolutional neural network trained to generate an illuminated image based on the amplified black-level-subtracted packed data, wherein the convolutional neural network is trained based on images captured by a sensor similar to the imaging sensor.

2. The apparatus of claim 1 , wherein the convolutional neural network is trained based on a training set comprising pairs of images captured using fast-exposure and long-exposure shutter speeds.

3. The apparatus of claim 1 , wherein the convolutional neural network is trained based on an L1 loss, an L2 loss, or a structural similarity (SSIM) loss.

4. The apparatus of claim 1 , wherein the convolutional neural network is trained using an Adam optimizer, a gradient decent optimizer, a proximal gradient descent optimizer, an RMSProp optimizer, a Momentum optimizer, or an Adadelta optimizer.

5. The apparatus of claim 1 , wherein the convolutional neural network is trained using an amplification ratio based on an exposure difference between an input training image and a corresponding reference image comprising an image captured using a longer shutter speed.

6. The apparatus of claim 1 , wherein the illuminated image comprises a 12-channel image having a spatial resolution comprising half the spatial resolution of the raw sensor data, the apparatus comprising a sub-pixel layer to process the illuminated image to generate the illuminated image comprising a resolution equal to the resolution of the raw sensor data.

7. The apparatus of claim 1 , wherein the convolutional neural network is trained for a specific camera sensor using a training set of images captured using the specific camera sensor.

8. The apparatus of claim 1 , wherein the convolutional neural network comprises a fully convolutional network (FCN).

9. The apparatus of claim 1 , wherein the convolutional neural network comprises a multi-scale context aggregation network.

10. The apparatus of claim 1 , wherein the convolutional neural network comprises a U-net architecture.

11. A method for low-light imaging, comprising:
receiving, via a pipelined processor, raw sensor data from an imaging sensor;
subtracting a black level from packed data to generate black-level-subtracted packed data, the black level generated from the raw sensor data;
amplifying the black-level-subtracted packed data based on a received amplification ratio to generate amplified black-level-subtracted packed data and sending the amplified black-level-subtracted packed data to a trained convolutional neural network; and
generating, via the trained convolutional neural network of the pipelined processor, an illuminated image based on the amplified black-level-subtracted packed data, wherein the trained convolutional neural network is trained based on images captured by a sensor similar to the imaging sensor.

12. The method of claim 11 , comprising packing the raw sensor data into four channels comprising a reduced spatial resolution by a factor of two in each direction to generate packed data and inputting the packed data into the trained convolutional neural network, wherein the raw sensor data comprises a Bayer array.

13. The method of claim 11 , comprising arranging the raw sensor data into 6×6 blocks and pack the raw sensor data into nine channels by exchanging adjacent elements of the 6×6 blocks to generate packed data and inputting the packed data into the trained convolutional neural network, wherein the raw sensor data comprises an X-Trans array.

14. The method of claim 11 , comprising processing the illuminated image to generate the illuminated image comprising a resolution equal to the resolution of the raw sensor data.

15. The method of claim 11 , comprising performing, via the trained convolutional neural network, blind noise suppression on the raw sensor data.

16. The method of claim 11 , comprising performing, via the trained convolutional neural network, a color transformation on the raw sensor data.

17. The method of claim 11 , comprising processing the illuminated image using histogram stretching.

18. The method of claim 11 , comprising processing the illuminated image using high dynamic resolution (HDR) tone mapping.

19. At least one computer readable medium for imaging in low-light environments having instructions stored therein that, in response to being executed on a computing device, cause the computing device to:
receive raw sensor data from an imaging sensor; and
generate a black level from the raw sensor data and subtract the black level from packed data to generate black-level-subtracted packed data;
amplify the black-level-subtracted packed data based on a received amplification ratio to generate amplified black-level-subtracted packed data to be input into a convolutional neural network; and
generate, via the convolutional neural network trained using a training set of images captured using a sensor of a similar type as the imaging sensor, an illuminated image based on the amplified black-level-subtracted packed data.

20. The at least one computer readable medium of claim 19 , comprising instructions to pack the raw sensor data into four channels comprising a reduced spatial resolution by a factor of two in each direction to generate packed data, wherein the raw sensor data comprises a Bayer array.

21. The at least one computer readable medium of claim 19 , comprising instructions to arrange the raw sensor data into 6×6 blocks and pack the raw sensor data into nine channels by exchanging adjacent elements of the 6×6 blocks to generate packed data, wherein the raw sensor data comprises an X-Trans array.

Assignees

• Intel Corp

Inventors

• Chen Chen
• Chen Qifeng
• Koltun Vladlen

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T5/40

  using histogram techniques · CPC title