Dual sensor imaging system and privacy protection imaging method thereof

Patent metadata

Field	Value
Publication number	US-11689822-B2
Application number	US-202117214937-A
Country	US
Kind code	B2
Filing date	Mar 29, 2021
Priority date	Sep 4, 2020
Publication date	Jun 27, 2023
Grant date	Jun 27, 2023

Abstract

A dual sensor imaging system and a privacy protection imaging method thereof are provided. The system is configured to control at least one color sensor and at least one IR sensor to respectively capture multiple color images and multiple IR images by adopting multiple exposure conditions adapted for an imaging scene, adaptively select a combination of the color image and the IR image that can reveal details of the imaging scene, detect a feature area with features of a target of interest in the color image, and fuse the color image and the IR image to generate a fusion image with the details of the imaging scene, and crop an image of the feature area of the fusion image to be replaced with an image not belonging to the IR image, so as to generate a scene image.

First claim

What is claimed is:

1. A dual sensor imaging system, comprising:
at least one color sensor;
at least one infrared ray (IR) sensor;
a storage device, storing a computer program; and
a processer, coupled to the at least one color sensor, the at least one IR sensor, and the storage device, and configured to load and execute the computer program to:
control the at least one color sensor and the at least one IR sensor to respectively capture a plurality of color images and a plurality of IR images by adopting a plurality of exposure conditions adapted for an imaging scene;
adaptively select a combination of the color image and the IR image that can reveal details of the imaging scene;
detect a feature area with at least one feature of a target of interest in the color image according to the at least one feature; and
fuse the selected color image and IR image to generate a fusion image with the details of the imaging scene, and crop an image of the feature area in the fusion image to be replaced with an image not belonging to the IR image, so as to generate a scene image;

wherein the processor:
selects one of the color images as a reference image according to color details of the color image;
identifies at least one defect area lacking texture details in the reference image; and
selects one of the IR images as the IR image for fusion with the reference image according to texture details of an image corresponding to the at least one defect area in each of the IR images.

2. The dual sensor imaging system according to claim 1 , wherein the processor further:
controls the at least one color sensor to capture a plurality of color images by adopting a plurality of exposure times longer or shorter than an exposure time of the selected color image, and executes a high dynamic range (HDR) process to generate an HDR image with details of the feature area to be used to replace an image of the feature area in the cropped fusion image.

3. The dual sensor imaging system according to claim 2 , wherein the processor:
selects the exposure time for capturing the color images according to the details of the feature area of the color image, so that the HDR image with the details of the feature region is generated after the captured color images are processed by the HDR process.

4. The dual sensor imaging system according to claim 1 , wherein the processor:
controls at least one of the at least one color sensor and the at least one IR sensor to capture at least one standard image of the imaging scene by adopting a standard exposure condition, and identifies the imaging scene using the at least one standard image.

5. The dual sensor imaging system according to claim 1 , wherein the processor:
selects the color image with the most color details as the reference; and
selects the IR image with the most texture details of the image corresponding to the at least one defect area as the IR image for fusion with the reference image.

6. The dual sensor imaging system according to claim 1 , wherein the processor:
replaces a brightness component of an image of the at least one defect area in the reference image with the image corresponding to the at least one defect area in the IR image to generate the scene image that complements the texture details of the at least one defect area.

7. The dual sensor imaging system according to claim 1 , wherein the processor further:
determines whether each of the IR images comprises the texture details of the at least one defect area; and
replaces an image of the at least one defect area in the reference image with an image corresponding to the at least one defect area in the HDR image when none of the IR images comprises the texture details to generate the scene image with the texture details of the at least one defect area.

8. The dual sensor imaging system according to claim 1 , wherein
the processor further:
uses a machine learning model to identify the target of interest in the color image, so as to detect the feature area, wherein the machine learning model is trained by using a plurality of color images comprising the target of interest and an identification result of the target of interest in each of the color images.

9. The dual sensor imaging system according to claim 8 , wherein the machine learning model comprises an input layer, at least one hidden layer, and an output layer, and the processor:
sequentially inputs the color images to the input layer, wherein a plurality of neurons in each of the at least one hidden layer uses an activation function to calculate a current output for an output of the input layer, and the output layer converts the output of the hidden layer into a prediction result of the target of interest; and
compares the prediction result with the identification result corresponding to the currently input color image to update a weight of each of the neurons of the hidden layer according to a comparison result; and
repeats the above steps to train the machine learning model to identify the target of interest.

10. A privacy protection imaging method of a dual sensor imaging system, wherein the dual sensor imaging system comprises at least one color sensor, at least one IR sensor, and a processor, the privacy protection imaging method comprising:
controlling the at least one color sensor and the at least one IR sensor to respectively capture a plurality of color images and a plurality of IR images by adopting a plurality of exposure conditions adapted for an imaging scene;
adaptively selecting a combination of the color image and the IR image that can reveal details of the imaging scene;
detecting a feature area with at least one feature of a target of interest in the selected color image according to the at least one feature; and
fusing the selected color image and IR image to generate a fusion image with the details of the imaging scene, and cropping an image of the feature area in the fusion image to be replaced with an image not belonging to the IR image, so as to generate a scene image;
wherein the step of adaptively selecting the combination of the color image and the IR image that can reveal the details of the imaging scene comprises:
selecting one of the color images as a reference image according to color details of the color image;
identifying at least one defect area lacking texture details in the reference image; and
selecting one of the IR images as the IR image for fusion with the reference image according to texture details of an image corresponding to the at least one defect area in each of the IR images.

11. The privacy protection imaging method according to claim 10 , further comprising:
controlling the at least one color sensor to capture a plurality of color images by adopting a plurality of exposure times longer or shorter than an exposure time of the selected color image, and executing an HDR process to generate an HDR image with details of the feature area, and using the HDR image to replace the image of the feature area in the cropped fusion image.

12. The privacy protection imaging method according to claim 10 , wherein the step of identifying the imaging scene of the dual sensor imaging system comprises:
controlling at least one of the at least one color sensor and the at least one IR sensor to capture at least one standard image of the imaging scene by adopting a standard exposure condition, and identifying the imaging scene using the at least one standard image.

13. The privacy protection imaging method according to claim 10 , wherein the step of adaptively selecting the combination of the color image and the IR image that can reveal the details of the

Assignees

• Altek Semiconductor Corp

Inventors

• Peng Shih-Yuan
• Cheng shu-chun
• Huang Hsu-Lien
• Li Yun-Chin
• Lai Kuo-Ming

Classifications

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N9/04559 Primary

  Electricity · mapped topic

• H04N5/772 Primary

  the recording apparatus and the television camera being placed in the same enclosure · CPC title

• G06T2207/30168

  Image quality inspection · CPC title