Method and apparatus with image fusion

Patent metadata

Field	Value
Publication number	US-11449971-B2
Application number	US-201816223382-A
Country	US
Kind code	B2
Filing date	Dec 18, 2018
Priority date	Feb 9, 2018
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

Disclosed is an image fusion method and apparatus. The fusion method includes detecting first feature points of an object in a first image frame from the first image frame; transforming the first image frame based on the detected first feature points and predefined reference points to generate a transformed first image frame; detecting second feature points of the object in a second image frame from the second image frame; transforming the second image frame based on the detected second feature points and the predefined reference points to generate a transformed second image frame; and generating a combined image by combining the transformed first image frame and the transformed second image frame.

First claim

What is claimed is:

1. A processor implemented image fusion method, comprising:
detecting first feature points of an object in a first image frame from the first image frame;
transforming the first image frame based on the detected first feature points and predefined reference points to generate a transformed first image frame;
detecting second feature points of the object in a second image frame from the second image frame;
transforming the second image frame based on the detected second feature points and the predefined reference points to generate a transformed second image frame;
generating a combined image by combining the transformed first image frame and the transformed second image frame;
inverse transforming the combined image based on a correspondence between the second feature points and the predefined reference points;
generating a resulting image by applying the inverse transformed combined image to the second image frame; and
performing object verification or authentication of the object based on the resulting image.

2. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image fusion method of claim 1 .

3. The image fusion method of claim 1 , wherein the detecting of the first feature points comprises detecting facial landmarks from the first image frame, and the detecting of the second feature points comprises detecting facial landmarks from the second image frame.

4. The image fusion method of claim 1 ,
wherein a first operation, including the detection of the first feature points and the transforming of the first image frame, and the generating of the combined image are selectively performed based on a determining of whether a measured image quality of the first image frame or the transformed first image frame meets a preset threshold value, and
wherein a second operation, including the detection of the second feature points and the transforming of the second image frame, and the generating of the combined image are selectively performed based on a determining of whether a measured image quality of the second image frame or the transformed second image frame meets the preset threshold value.

5. The image fusion method of claim 4 , further comprising:
measuring the image quality of the first image frame or the transformed first image frame; and
measuring the image quality of the second image frame or the transformed second image frame,
wherein the measuring of the image quality of the first image frame and the measuring of the image quality of the second image frame respectively comprise measuring a brightness of the first image frame and measuring a brightness of the second image frame.

6. The image fusion method of claim 1 ,
wherein the transforming of the first image frame includes transforming the first image frame such that the first feature points are respectively placed at, or respectively warped with respect to, positions of the predefined reference points to generate the transformed first image frame, and
wherein the transforming of the second image frame includes transforming the second image frame such that the second feature points are respectively placed at, or respectively warped with respect to, positions of the predefined reference points to generate the transformed second image frame.

7. The image fusion method of claim 1 ,
wherein the transforming of the first image frame comprises:
detecting a first object region from the first image frame based on the detected first feature points; and
transforming the first object region through an image warping technique based on first feature points included in the detected first object region and the predefined reference points, and
wherein the transforming of the second image frame comprises:
detecting a second object region from the second image frame based on the detected second feature points; and
transforming the second object region through the image warping technique based on second feature points included in the detected second object region and the predefined reference points.

8. The image fusion method of claim 7 , wherein the generating of the combined image comprises combining the transformed first object region and the transformed second object region.

9. The image fusion method of claim 1 , wherein the generating of the combined image comprises combining a first pixel value of the transformed first image frame and a second pixel value of the transformed second image frame.

10. The image fusion method of claim 9 , wherein the combining of the first pixel value and the second pixel value includes any one or any combination of any two or more of a summing, an averaging, a weighted summing, and a weighted averaging of the first pixel value and the second pixel value at corresponding positions in the transformed first image frame and the transformed second image frame.

11. The image fusion method of claim 1 , further comprising:
detecting third feature points of the object shown in a third image frame from the third image frame;
transforming the third image frame based on the detected third feature points and the reference points; and
combining the obtained combination image and the transformed third image frame.

12. An image fusion apparatus, comprising:
a processor configured to:
detect first feature points of an object in a first image frame from the first image frame;
transform the first image frame based on the detected first feature points and predefined reference points to generate a transformed first image frame;
detect second feature points of the object in a second image frame from the second image frame;
transform the second image frame based on the detected second feature points and the predefined reference points to generate a transformed second image frame;
generate a combined image by combining the transformed first image frame and the transformed second image frame;
inverse transform the combined image based on a correspondence between the second feature points and the predefined reference points;
generate a resulting image by applying the inverse transformed combined image to the second image frame; and
perform object verification or authentication of the object based on the resulting image.

13. The image fusion apparatus of claim 12 ,
wherein the processor is configured to selectively perform a first operation, including the detection of the first feature points and the transforming of the first image frame, and the generating of the combined image based on a determining of whether a measured image quality of the first image frame or the transformed first image frame meets a preset threshold value, and
wherein the processor is configured to selectively perform a second operation, including the detection of the second feature points and the transforming of the second image frame, and the generating of the combined image based on a determining of whether a measured image quality of the second image frame or the transformed second image frame meets the preset threshold value.

14. The image fusion apparatus of claim 13 , wherein the processor is further configured to:
measure the image quality of the first image frame or the transformed first image frame; and
measure the image quality of the second image frame or the transformed second image frame,
wherein the measuring of the image quality of the first image frame and the measuring of the image quality of the second image frame respectively comprise measuring a brightness of the first image frame and measuring a brightness of the second image frame.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Ko Minsu
• Han Seungju
• Han Jaejoon
• Kim Jihye
• Park Sungun
• Choi Chang Kyu

Classifications

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06T2207/30201

  Face · CPC title

• G06T2207/30168

  Image quality inspection · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title