Training image signal processors using intermediate loss functions

Patent metadata

Field	Value
Publication number	US-10796200-B2
Application number	US-201815965158-A
Country	US
Kind code	B2
Filing date	Apr 27, 2018
Priority date	Apr 27, 2018
Publication date	Oct 6, 2020
Grant date	Oct 6, 2020

Abstract

In an example method for training image signal processors, a reconstructed image is generated via an image signal processor based on a sensor image. An intermediate loss function is generated based on a comparison of an output of one or more corresponding layers of a computer vision network and a copy of the computer vision network. The output of the computer vision network is based on the reconstructed image. An image signal processor is trained based on the intermediate loss function.

First claim

What is claimed is:

1. An apparatus for training image signal processors, comprising:
an image signal processor to be trained, the image signal processor to generate a reconstructed image based on a sensor image;
an intermediate loss function generator to generate an intermediate loss function based on a comparison of intermediate outputs of one or more corresponding intermediate layers of a computer vision network and a copy of the computer vision network, wherein the computer vision network generates an intermediate output based on the reconstructed image and the copy of the computer vision network generates an intermediate output at a corresponding intermediate layer based on an image from a dataset used to generate the sensor image; and
a parameter modifier to modify one or more parameters of the image signal processor based on the intermediate loss function.

2. The apparatus of claim 1 , wherein the sensor image is received from a raw sensor dataset.

3. The apparatus of claim 1 , comprising an image sensor modeler to generate the sensor image based on the image from the dataset.

4. The apparatus of claim 1 , wherein the image signal processor comprises a deep learning image signal processor network.

5. The apparatus of claim 1 , wherein the computer vision network and the copy of the computer vision network comprise trained classifiers.

6. The apparatus of claim 1 , wherein the parameter modifier is to also modify one or more parameters of the computer vision network and the copy of the computer vision network.

7. The apparatus of claim 1 , wherein the comparison is to be performed by a deep learning network trained on a number of computer vision tasks.

8. The apparatus of claim 1 , wherein the intermediate output of the copy of the computer vision network is based on an ideal reconstructed image generated by processing the sensor image via an ideal image signal processor model.

9. The apparatus of claim 1 , comprising:
a loss function generator to generate a first loss function based on a ground truth and a final output of the computer vision network; and
a total loss function generator to generate a total loss function based on the first loss function and the intermediate loss function.

10. The apparatus of claim 9 , wherein the total loss function is based on a weighted combination of the first loss function and the intermediate loss function.

11. The apparatus of claim 10 , wherein the intermediate loss function is weighted higher in earlier iterations of training.

12. A method for training image signal processors, comprising:
generating, via an image signal processor, a reconstructed image based on a sensor image;
generating, via a processor, an intermediate loss function based on a comparison of intermediate outputs of one or more corresponding intermediate layers of a computer vision network and a copy of the computer vision network, wherein the output of the computer vision network is based on the reconstructed image, wherein the computer vision network generates an intermediate output based on the reconstructed image and the copy of the computer vision network generates an intermediate output at a corresponding intermediate layer based on an image from a dataset used to generate the sensor image; and
training, via the processor, an image signal processor based on the intermediate loss function.

13. The method of claim 12 , comprising generating, via the processor, the sensor image based on the image received from the dataset.

14. The method of claim 12 , comprising:
generating, via the processor, a first loss function based on a ground truth and a final output of the computer vision network;
generating, via the processor, a total loss function based on the first loss function and the intermediate loss function; and
training, via the processor, the image signal processor based on the total loss function.

15. The method of claim 12 , wherein training the image signal processor comprises reducing a weighting factor of the intermediate loss function based on a predetermined number of iterations.

16. The method of claim 12 , wherein training the image signal processor comprises modifying one or more parameters of the image signal processor.

17. The method of claim 12 , wherein training the image signal processor comprises auto-tuning one or more ISP parameters based on a feature map similarity.

18. The method of claim 12 , comprising training the computer vision network and the copy of the computer vision network.

19. The method of claim 12 , comprising performing the comparison via a deep learning network trained on a number of computer vision tasks.

20. The method of claim 12 , comprising generating, via an ideal image signal processor model, an ideal reconstructed image based on the sensor image, wherein the output of the copy of the computer vision network is based on the ideal reconstructed image.

21. At least one non-transitory computer readable medium for training image signal processors having instructions stored therein that, in response to being executed on a computing device, cause the computing device to:
generate a reconstructed image based on a sensor image;
generate an intermediate loss function based on a comparison of intermediate outputs of one or more corresponding intermediate layers of a computer vision network and a copy of the computer vision network, wherein the output of the computer vision network is based on the reconstructed image, wherein the computer vision network generates an intermediate output based on the reconstructed image and the copy of the computer vision network generates an intermediate output at a corresponding intermediate layer based on an image from a dataset used to generate the sensor image; and
train an image signal processor based on the intermediate loss function.

22. The at least one non-transitory computer readable medium of claim 21 , comprising instructions to generate the sensor image based on the image received from the dataset.

23. The at least one non-transitory computer readable medium of claim 21 , comprising instructions to:
generate a first loss function based on a ground truth and a final output of the computer vision network;
generate a total loss function based on the first loss function and the intermediate loss function; and
train the image signal processor based on the total loss function.

24. The at least one non-transitory computer readable medium of claim 21 , comprising instructions to modify one or more parameters of the computer vision network and the copy of the computer vision network.

25. The at least one non-transitory computer readable medium of claim 21 , comprising instructions to auto-tune one or more ISP parameters based on a feature map similarity.

Assignees

• Intel Corp

Inventors

• Sutic Aleksandar
• Zivkovic Zoran
• Michael Gilad

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06F18/214 Primary

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title