Predicting optimal values for parameters used in an operation of an image signal processor using machine learning

Patent metadata

Field	Value
Publication number	US-12437201-B2
Application number	US-202218067298-A
Country	US
Kind code	B2
Filing date	Dec 16, 2022
Priority date	May 21, 2019
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

A method of predicting optimal values for a plurality of parameters used in an operation of an image signal processor includes: inputting initial values for the plurality of parameters to a machine learning model having an input layer, corresponding to the plurality of parameters, and an output layer corresponding to a plurality of evaluation items extracted from a result image generated by the image signal processor; obtaining evaluation scores for the plurality of evaluation items using an output of the machine learning model; adjusting weights, applied to the plurality of parameters, based on the evaluation scores; and determining the optimal values using the adjusted weights.

First claim

What is claimed is:

1. A method of optimizing parameters for an image signal processor (ISP), comprising:
receiving sample data from an image sensor;
generating a plurality of different sample sets, each of the plurality of different sample sets including sample values for a plurality of parameters;
generating sample images, by an ISP simulator emulating the ISP processing the sample data based on each of the plurality of different sample sets;
evaluating each of the sample images, by an evaluation framework, to generate a plurality of sample score sets, each of the plurality of sample score sets including at least two sample scores;
training a machine learning model using each of the plurality of different sample sets as an input and adjusting weights for hidden layers of the machine learning model by comparing an output of the machine learning model and the at least two sample scores included in each of the plurality of sample score sets;
setting initial values for parameters for the ISP and initial weights for the parameters for the ISP;
applying the initial weights to the initial values to generate weighted initial values for the parameters;
inputting the weighted initial values to the trained machine learning model;
adjusting the initial weights by comparing outputs of the trained machine learning model and predetermined reference scores;
generating optimized parameters for the ISP based on the adjusted initial weights.

2. The method of claim 1 , wherein the at least two sample scores include scores for at least two of resolution, texture loss, sharpness, noise, a dynamic range, shading, and a color for each of the sample images.

3. The method of claim 1 , wherein the predetermined reference scores are adjusted by a user of a system including the ISP.

4. The method of claim 1 , wherein the initial values are generated at random.

5. The method of claim 1 , wherein the initial weights are adjusted until a difference between the predetermined reference scores and the outputs of the trained machine learning model is equal to or less than a reference difference.

6. The method of claim 1 , wherein the initial weights are adjusted for a predetermined number of times.

7. The method of claim 1 , wherein the optimized parameters are generated by applying the adjusted initial weights to the initial values.

8. An electronic system, comprising:
a simulator including a parameter generator to receive a sample data and generate a plurality of different sample sets and an image signal processor (ISP) simulator to generate sample images based on the plurality of different sample sets, each of the plurality of different sample sets including sample values for a plurality of parameters of the ISP simulator;
an evaluation framework configured to generate a plurality of sample score sets for the sample images;
a machine learning model trainer configured to train a machine learning model by using each of the plurality of different sample sets as an input and adjusting weights included in hidden layers of the machine learning model by comparing an output of the machine learning model and the plurality of sample score sets; and
an ISP parameter adjusting module configured to generate weighted initial parameters by applying initial weights to initial values for parameters of an ISP which is emulated by the ISP simulator, output the weighted initial parameters to the trained machine learning model,
wherein the ISP parameter adjusting module adjusts the initial weights based on a result of comparing outputs of the trained machine learning model and predetermined reference scores, and optimizes the parameters of the ISP based on the adjusted initial weights.

9. The electronic system of claim 8 , wherein each of the plurality of sample score sets includes at least two sample scores, and the at least two sample scores include scores for at least two of resolution, texture loss, sharpness, noise, a dynamic range, shading, and a color for each of the sample images.

10. The electronic system of claim 8 , wherein the ISP parameter adjusting module changes the parameters of the ISP, by adjusting the initial weights by referring to feedback from a user of the electronic system.

11. The electronic system of claim 10 , wherein the predetermined reference scores are adjusted by the user.

12. The electronic system of claim 8 , wherein the ISP parameter adjusting module adjusts the initial weights for a predetermined number of times.

13. The electronic system of claim 8 , wherein the ISP parameter adjusting module adjusts the initial weights until a difference between the predetermined reference scores and the output of the trained machine learning model is equal to or less than a reference difference.

14. The electronic system of claim 8 , further comprising a feedback module that compares the outputs of the trained machine learning model and the predetermined reference scores and transmits the result of comparing.

15. The electronic system of claim 8 , further comprising the ISP.

16. The electronic system of claim 15 , wherein a result image generated by the ISP is changed by adjusting the initial weights applied to the initial values for the parameters of the ISP.

17. The electronic system of claim 8 , wherein the ISP parameter adjusting module adjusts the initial weights applied to the initial values for the parameters of the ISP, and does not adjust the initial values for the parameters of the ISP.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Younghoon
• Kim Sungsu
• Lee Jungmin

Classifications

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G06T5/92

  based on global image properties · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title