Training image signal processors using intermediate loss functions

Patent metadata

Field	Value
Publication number	US-11386293-B2
Application number	US-202017063414-A
Country	US
Kind code	B2
Filing date	Oct 5, 2020
Priority date	Apr 27, 2018
Publication date	Jul 12, 2022
Grant date	Jul 12, 2022

Abstract

In an example method for training image signal processors, a reconstructed image is generated via an image signal processor based on a sensor image. An intermediate loss function is generated based on a comparison of an output of one or more corresponding layers of a computer vision network and a copy of the computer vision network. The output of the computer vision network is based on the reconstructed image. An image signal processor is trained based on the intermediate loss function.

First claim

What is claimed is:

1. An apparatus comprising:
an image signal processor to generate a reconstructed image based on a sensor image;
a first generator to implement a first loss function based on comparison of an intermediate layer of a first computer vision network with a corresponding intermediate layer of a second computer vision network, the first computer vision network to process the reconstructed image, the second computer vision network to process an input image associated with the sensor image;
a second generator to iteratively determine values of a second loss function by combining weighted values of the first loss function and a third loss function at successive training iterations, the third loss function based on one or more results output from the first computer vision network, the second generator to weight a value of the first loss function by a first weight at a first training iteration and weight a value of the third loss function by a second weight at the first training iteration; and
a parameter modifier to modify a parameter of the image signal processor based on at least one of the first loss function or the second loss function.

2. The apparatus of claim 1 , wherein the second computer vision network is initialized to be a copy of the first computer vision network.

3. The apparatus of claim 1 , further including a modeler to apply at least one of a color filter array, a point spread function or noise to the input image to generate the sensor image.

4. The apparatus of claim 1 , wherein the comparison is based on a mean square difference between a first set of results from the intermediate layer of the first computer vision network and a corresponding second set of results from the corresponding intermediate layer of the second computer vision network.

5. The apparatus of claim 1 , wherein the second generator is to:
decrease the first weight after a first number of training iterations; and
increase the second weight after the first number of training iterations.

6. At least one memory device comprising computer readable instructions that, when executed, cause processor circuitry to at least:
apply a sensor image to an image signal processor to generate a reconstructed image;
implement a first loss function based on comparison an intermediate layer of a first computer vision network with a corresponding intermediate layer of a second computer vision network, the first computer vision network to process the reconstructed image, the second computer vision network to process an input image associated with the sensor image;
iteratively determine values of a second loss function by combining weighted values of the first loss function and a third loss function at successive training iterations, the third loss function based on one or more results output from the first computer vision network, the instructions to cause the processor circuitry to weight a value of the first loss function by a first weight at a first training iteration and weight a value of the third loss function by a second weight at the first training iteration; and
modify a parameter of the image signal processor based on at least one of the first loss function or the second loss function.

7. The at least one memory device of claim 6 , wherein the instructions cause the processor circuitry to initialize the second computer vision network to be a copy of the first computer vision network.

8. The at least one memory device of claim 6 , wherein the instructions cause the processor circuitry to apply at least one of a color filter array, a point spread function or noise to the input image to generate the sensor image.

9. The at least one memory device of claim 6 , wherein the comparison is based on a mean square difference between a first set of results from the intermediate layer of the first computer vision network and a corresponding second set of results from the corresponding intermediate layer of the second computer vision network.

10. The at least one memory device of claim 6 , wherein the instructions cause the processor circuitry to:
decrease the first weight after a first number of training iterations; and
increase the second weight after the first number of training iterations.

11. A method comprising:
generating, with an image signal processor, a reconstructed image from a sensor image;
determining, by executing an instruction with processor circuitry, values of a first loss function at successive training iterations based on comparison of an intermediate layer of a first computer vision network with a corresponding intermediate layer of a second computer vision network, the first computer vision network to process the reconstructed image, the second computer vision network to process an input image associated with the sensor image;
iteratively determining, by executing an instruction with the processor circuitry, values of a second loss function by combining weighted values of the first loss function and a third loss function at successive training iterations, the third loss function based on one or more results output from the first computer vision network, a value of the first loss function weighted by a first weight at a first training iteration and a value of the third loss function weighted by a second weight at the first training iteration; and
modifying, by executing an instruction with the processor circuitry, a parameter of the image signal processor based on at least one of the first loss function or the second loss function.

12. The method of claim 11 , further including:
initializing the second computer vision network to be a copy of the first computer vision network; and
applying at least one of a color filter array, a point spread function or noise to the input image to generate the sensor image.

13. The method of claim 11 , wherein the comparison is based on a mean square difference between a first set of results from the intermediate layer of the first computer vision network and a corresponding second set of results from the corresponding intermediate layer of the second computer vision network.

14. The method of claim 11 , further including:
decreasing the first weight after a first number of training iterations; and
increasing the second weight after the first number of training iterations.

Assignees

• Intel Corp

Inventors

• Sutic Aleksandar
• Zivkovic Zoran
• Michael Gilad

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/809

  of classification results, e.g. where the classifiers operate on the same input data · CPC title

• G06F18/214 Primary

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title