Calibration method for three-dimensional (3D) augmented reality and apparatus thereof

Patent metadata

Field	Value
Publication number	US-10928901-B2
Application number	US-201916400100-A
Country	US
Kind code	B2
Filing date	May 1, 2019
Priority date	Dec 4, 2018
Publication date	Feb 23, 2021
Grant date	Feb 23, 2021

Abstract

A calibration method for a three-dimensional (3D) augmented reality and an apparatus thereof are provided. The calibration method includes determining a first conversion parameter representing a relationship between a coordinate system of an eye-tracking camera and a coordinate system of a calibration camera by capturing a physical pattern using the eye-tracking camera and the calibration camera, and determining a second conversion parameter representing a relationship between a coordinate system of a virtual screen and the coordinate system of the calibration camera and a size parameter representing a size of the virtual screen by capturing a virtual pattern displayed on the virtual screen using the calibration camera.

First claim

What is claimed is:

1. A calibration method for a three-dimensional (3D) augmented reality, the method comprising:
determining a first conversion parameter representing a first relationship between a first coordinate system of an eye-tracking camera and a second coordinate system of a calibration camera by capturing a physical pattern using the eye-tracking camera and the calibration camera;
determining a second conversion parameter representing a second relationship between a third coordinate system of a virtual screen and the second coordinate system of the calibration camera and a size parameter representing a size of the virtual screen by capturing a virtual pattern displayed on the virtual screen using the calibration camera; and
estimating a conversion relationship between the first coordinate system of the eye-tracking camera and the third coordinate system of the virtual screen based on the first conversion parameter, the second conversion parameter, and the size parameter.

2. The calibration method of claim 1 , wherein the determining of the first conversion parameter comprises:
determining a third conversion parameter representing a third relationship between the first coordinate system of the eye-tracking camera and a fourth coordinate system of the physical pattern based on an image acquired by the eye-tracking camera from capturing a reflector illuminating the physical pattern;
determining a fourth conversion parameter representing a fourth relationship between the second coordinate system of the calibration camera and the fourth coordinate system of the physical pattern based on an image acquired by the calibration camera from capturing the physical pattern; and
determining the first conversion parameter based on the third conversion parameter between the first coordinate system of the eye-tracking camera and the fourth coordinate system of the physical pattern and the fourth conversion parameter between the second coordinate system of the calibration camera and the fourth coordinate system of the physical pattern.

3. The calibration method of claim 1 , wherein the determining of the second conversion parameter and the size parameter comprises:
estimating positions of feature points included in the virtual pattern based on images acquired by the calibration camera capturing the virtual pattern; and
determining the second conversion parameter and the size parameter based on the estimated positions of the feature points.

4. The calibration method of claim 1 , wherein the determining of the second conversion parameter and the size parameter comprises:
estimating positions of feature points included in the virtual pattern based on images acquired by the calibration camera capturing the virtual pattern;
parameterizing the positions of the feature points based on the third coordinate system of the virtual screen; and
determining the second conversion parameter and the size parameter by minimizing differences between the estimated positions of the feature points and the parameterized positions of the feature points.

5. The calibration method of claim 4 , wherein the estimating of the positions of the feature points comprises:
estimating 3D positions of the feature points based on two-dimensional (2D) positions of the feature points appearing in an image acquired by a first sub-camera of the calibration camera capturing the virtual pattern, 2D positions of the feature points appearing in an image acquired by a second sub-camera of the calibration camera capturing the virtual pattern, and a conversion parameter between a fifth coordinate system of the first sub-camera and a sixth coordinate system of the second sub-camera.

6. The calibration method of claim 4 , wherein the parameterizing of the positions of the feature points comprises:
parameterizing the positions of the feature points based on a size of a unit cell of the virtual pattern.

7. The calibration method of claim 1 , wherein the size parameter includes values representing leftmost, rightmost, uppermost, and lowermost boundaries of the virtual screen.

8. The calibration method of claim 1 , further comprising:
determining intrinsic parameters of the eye-tracking camera and the calibration camera and a conversion parameter between a fifth coordinate system of a first sub-camera of the calibration camera and a sixth coordinate system of a second sub-camera of the calibration camera.

9. The calibration method of claim 1 , wherein an image for a 3D augmented reality is rendered based on the conversion relationship between the first coordinate system of the eye-tracking camera and the third coordinate system of the virtual screen.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the calibration method of claim 1 .

11. A calibration apparatus for a three-dimensional (3D) augmented reality, the apparatus comprising:
a memory configured to store one or more instructions; and
a processor, when the one or more instructions are executed, configured to:
determine a first conversion parameter representing a first relationship between a first coordinate system of an eye-tracking camera and a second coordinate system of a calibration camera by capturing a physical pattern using the eye-tracking camera and the calibration camera;

determine a second conversion parameter representing a second relationship between a third coordinate system of a virtual screen and the second coordinate system of the calibration camera and a size parameter representing a size of the virtual screen by capturing a virtual pattern displayed on the virtual screen using the calibration camera; and
estimate a conversion relationship between the first coordinate system of the eye-tracking camera and the third coordinate system of the virtual screen based on the first conversion parameter, the second conversion parameter, and the size parameter.

12. The calibration apparatus of claim 11 , wherein the processor is further configured to:
determine a third conversion parameter representing a third relationship between the first coordinate system of the eye-tracking camera and a fourth coordinate system of the physical pattern based on an image acquired by the eye-tracking camera from capturing a reflector illuminating the physical pattern;
determine a fourth conversion parameter representing a fourth relationship between the second coordinate system of the calibration camera and the fourth coordinate system of the physical pattern based on an image acquired by the calibration camera from capturing the physical pattern; and
determine the first conversion parameter based on the third conversion parameter between the first coordinate system of the eye-tracking camera and the fourth coordinate system of the physical pattern and the fourth conversion parameter between the second coordinate system of the calibration camera and the fourth coordinate system of the physical pattern.

13. The calibration apparatus of claim 11 , wherein the processor is further configured to:
estimate positions of feature points included in the virtual pattern based on images acquired by the calibration camera capturing the virtual pattern; and
determine the second conversion parameter and the size parameter based on the estimated positions of the feature points.

14. The calibration apparatus of claim 11 , wherein the processor is further configured to:
estimate positions of feature points included in the virtual pattern based on images acquired by the calibration camera capturing the virtual pattern;
parameterize the position

Assignees

• Samsung Electronics Co Ltd

Inventors

• Park Juyong
• Lee Seok

Classifications

• G06T7/85 Primary

  Stereo camera calibration · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T2207/30268

  Vehicle interior · CPC title