Image merging method using viewpoint transformation and system therefor

Patent metadata

Field	Value
Publication number	US-10972712-B2
Application number	US-201816201906-A
Country	US
Kind code	B2
Filing date	Nov 27, 2018
Priority date	Nov 29, 2017
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

An image merging method using viewpoint transformation and a system therefor are provided. The method includes obtaining images captured by a plurality of cameras included in the camera system, performing viewpoint transformation for each of the images using a depth map for the images, and merging the images, the viewpoint transformation of which is performed.

First claim

What is claimed is:

1. An image merging method using viewpoint transformation executed by a computer included in a camera system including a plurality of cameras, the method comprising:
performing, by the camera system, viewpoint transformation for each image obtained by the plurality of cameras, using a depth map; and
merging images, the viewpoint transformation of which is performed;
wherein said performing the viewpoint transformation for each of the images comprises:
setting a viewpoint transformation relationship including a backward warping relationship;
determining a movement parameter in the viewpoint transformation relationship, the movement parameter being a value associated with a distance to move when a location at which each of the images is captured moves to a virtual specific location; and
transforming viewpoints of the images to be matched using the viewpoint transformation relationship;

wherein the backward warping is performed in accordance with the following relationship:

I
⁢

(

x
,
y

)

=

I
′

⁡

(

x
′

⁡

(

fB
+

m
z

⁢
D

)

+

f
⁢

m
x

⁢
D

fB

-

c

x
′

⁢

y
′

⁡

(

fB
+

m
y

⁢
D

)

+

f
⁢

m
z

⁢
D

fB

-

c
y

)

in which I(x, y) denotes the images captured by the plurality of cameras included in the camera system, x, y denote pixel locations in each of the images captured by the plurality of cameras included in the camera system, I′(x′, y′) denotes transformed images resulting from the viewpoint relationship which is performed, x′, y′ denote pixel locations in each of the transformed images resulting from the viewpoint transformation which is performed, f denotes a focal length of each of the plurality of cameras, B denotes a length of a baseline of the camera system, D denotes a parallax between the plurality of cameras, m x , m y , m denote the movement parameter, and c x , c y denote coordinates of an image center passing through an optical axes.

2. The method of claim 1 , wherein the determining of the movement parameter in the viewpoint transformation formula comprises:
determining the movement parameter based on a distance to move when an optical center of each of the plurality of cameras moves to a virtual reference location of the camera system.

3. The method of claim 2 , wherein said determining of the movement parameter in the viewpoint transformation relationship comprises:
determining the movement parameter based on a length of a baseline of the camera system.

4. The method of claim 1 , wherein said transforming the viewpoints of the images to be matched comprises:
transforming the viewpoint of each of the images into a viewpoint of an image captured at a reference location of the camera system.

5. The method of claim 1 , wherein said performing the viewpoint transformation for each of the images further comprises:
filling a hole of the depth map before performing the viewpoint transformation for each of the images.

6. The method of claim 1 , wherein said performing the viewpoint transformation for each of the images further comprises:
filling a hole of each of the images, the viewpoint transformation of which is performed.

7. The method of claim 1 , wherein the merging of the images, the viewpoint transformation of which is performed, further comprises:
transforming a viewpoint of the merged image to fill a hole of the merged image.

8. The method of claim 1 , wherein obtaining the images by the plurality of cameras comprises:
obtaining the images with different exposure values from the plurality of cameras; and
wherein said merging the images, the viewpoint transformation of which is performed, comprises:
merging the images, with the different exposure values, the viewpoint transformation of which is performed, to generate a hyper dynamic range (HDR) image.

9. The method of claim 1 , wherein each image captured by the plurality of cameras is obtained by:
obtaining images with different focus positions from the plurality of cameras; and
wherein said merging the images, the viewpoint transformation of which is performed, comprises:
merging the images, with the different focus positions, the viewpoint transformation of which is performed, to generate an all-in-focus image.

10. An image merging system using viewpoint transformation, implemented with a computer included in a camera system, the system comprising:
at least one processor configured to execute computer-readable instructions,
wherein the at least one processor compri

Assignees

• Ct Integrated Smart Sensors Found

Inventors

• Park Ki Yeong

Classifications

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• H04N13/156

  Mixing image signals · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20208

  High dynamic range [HDR] image processing · CPC title