Synthetic data generation for machine learning-based post-processing

Patent metadata

Field	Value
Publication number	US-12541816-B2
Application number	US-202318363596-A
Country	US
Kind code	B2
Filing date	Aug 1, 2023
Priority date	Aug 1, 2023
Publication date	Feb 3, 2026
Grant date	Feb 3, 2026

Abstract

A method includes obtaining a ground truth image and generating multiple image frames using the ground truth image, a modeled optical blur, and a modeled global motion. The method also includes generating multiple mosaic image frames using the image frames and a color filter array and generating multiple raw input image frames using the mosaic image frames and a noise model associated with at least one imaging sensor. The method further includes providing the raw input image frames to a multi-frame processing pipeline in order to generate synthetic training data. In addition, the method includes training a machine learning-based image processing engine using the ground truth image and the synthetic training data.

First claim

What is claimed is:

1 . A method comprising:
obtaining a ground truth image; generating multiple image frames using the ground truth image, a modeled optical blur, and a modeled global motion; generating multiple mosaic image frames using the image frames and a color filter array; generating multiple raw input image frames using the mosaic image frames and a noise model associated with at least one imaging sensor; providing the raw input image frames to a multi-frame processing pipeline in order to generate synthetic training data; and training a machine learning-based image processing engine using the ground truth image and the synthetic training data.

2 . The method of claim 1 , wherein generating the image frames comprises:
applying the modeled optical blur to the ground truth image in order to generate a blurred image frame; and introducing different global motions to the blurred image frame in order to generate different ones of the image frames.

3 . The method of claim 2 , wherein applying the modeled optical blur to the ground truth image comprises:
applying an optics model to the ground truth image, the optics model representing the modeled optical blur that is associated with the at least one imaging sensor.

4 . The method of claim 1 , wherein generating the mosaic image frames comprises:
applying the color filter array to the image frames in order to convert the image frames into red-green-blue (RGB) mosaic image frames.

5 . The method of claim 1 , wherein generating the raw input image frames comprises:
applying noise to the mosaic image frames using the noise model based on one or more parameters of the at least one imaging sensor.

6 . The method of claim 1 , wherein:
the multi-frame processing pipeline comprises multiple stages; and providing the raw input image frames to the multi-frame processing pipeline in order to generate the synthetic training data comprises:
tapping a specified one of the stages in the multi-frame processing pipeline based on at least one task to be performed by the machine learning-based image processing engine; and
obtaining an input or an output of the specified stage as the synthetic training data.

7 . The method of claim 6 , wherein:
the multi-frame processing pipeline comprises a raw image frame processing stage, a multi-frame alignment stage, a demosaic stage, a tone-mapping stage, and a noise reduction stage; the specified stage represents one of the raw image frame processing stage, the multi-frame alignment stage, the demosaic stage, the tone-mapping stage, and the noise reduction stage; and training the machine learning-based image processing engine comprises training the machine learning-based image processing engine to perform one or more tasks associated with one or more of the stages in the multi-frame processing pipeline that follow the specified stage.

8 . An electronic device comprising:
at least one processing device configured to:
obtain a ground truth image;
generate multiple image frames using the ground truth image, a modeled optical blur, and a modeled global motion;
generate multiple mosaic image frames using the image frames and a color filter array;
generate multiple raw input image frames using the mosaic image frames and a noise model associated with at least one imaging sensor;
provide the raw input image frames to a multi-frame processing pipeline in order to generate synthetic training data; and
train a machine learning-based image processing engine using the ground truth image and the synthetic training data.

9 . The electronic device of claim 8 , wherein, to generate the image frames, the at least one processing device is configured to:
apply the modeled optical blur to the ground truth image in order to generate a blurred image frame; and introduce different global motions to the blurred image frame in order to generate different ones of the image frames.

10 . The electronic device of claim 9 , wherein, to apply the modeled optical blur to the ground truth image, the at least one processing device is configured to apply an optics model to the ground truth image, the optics model representing the modeled optical blur that is associated with the at least one imaging sensor.

11 . The electronic device of claim 8 , wherein, to generate the mosaic image frames, the at least one processing device is configured to apply the color filter array to the image frames in order to convert the image frames into red-green-blue (RGB) mosaic image frames.

12 . The electronic device of claim 8 , wherein, to generate the raw input image frames, the at least one processing device is configured to apply noise to the mosaic image frames using the noise model based on one or more parameters of the at least one imaging sensor.

13 . The electronic device of claim 8 , wherein:
the multi-frame processing pipeline comprises multiple stages; and to provide the raw input image frames to the multi-frame processing pipeline in order to generate the synthetic training data, the at least one processing device is configured to:
tap a specified one of the stages in the multi-frame processing pipeline based on at least one task to be performed by the machine learning-based image processing engine; and
obtain an input or an output of the specified stage as the synthetic training data.

14 . The electronic device of claim 13 , wherein:
the multi-frame processing pipeline comprises a raw image frame processing stage, a multi-frame alignment stage, a demosaic stage, a tone-mapping stage, and a noise reduction stage; the specified stage represents one of the raw image frame processing stage, the multi-frame alignment stage, the demosaic stage, the tone-mapping stage, and the noise reduction stage; and to train the machine learning-based image processing engine, the at least one processing device is configured to train the machine learning-based image processing engine to perform one or more tasks associated with one or more of the stages in the multi-frame processing pipeline that follow the specified stage.

15 . A method comprising:
obtaining multiple input image frames; and generating a blended image based on the input image frames using a machine learning-based image processing engine; wherein the machine learning-based image processing engine comprises a machine learning model that has been trained using a ground truth image and synthetic training data that is generated based on (i) the ground truth image, (ii) a modeled optical blur, (iii) a modeled global motion, (iv) a color filter array, and (v) a noise model associated with at least one imaging sensor.

16 . The method of claim 15 , wherein the machine learning model is trained by:
obtaining the ground truth image; generating multiple image frames using the ground truth image, the modeled optical blur, and the modeled global motion; generating multiple mosaic image frames using the image frames and the color filter array; generating multiple raw input image frames using the mosaic image frames and the noise model; providing the raw input image frames to a multi-frame processing pipeline in order to generate the synthetic training data; and training the machine learning-based image processing engine using the ground truth image and the synthetic training data.

17 . The method of claim 16 , wherein:
applying the modeled optical blur to the ground truth image comprises applying an optics model to the ground truth image, the optics model representing the modeled optical blur that is associa

Assignees

• Samsung Electronics Co Ltd

Inventors

• Gnanasambandam Abhiram
• Glotzbach John W
• Lee John Seokjun
• Sheikh Hamid R

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• H04N23/84

  for processing colour signals · CPC title

• G06T5/73

  Deblurring; Sharpening · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title