Localization method and system for augmented reality in mobile devices

Patent metadata

Field	Value
Publication number	US-10997744-B2
Application number	US-201916374106-A
Country	US
Kind code	B2
Filing date	Apr 3, 2019
Priority date	Apr 3, 2018
Publication date	May 4, 2021
Grant date	May 4, 2021

Abstract

The present invention relates to localization method and system for providing augmented reality in mobile devices and includes sub-sampling image data acquired from a camera in the mobile devices, and extracting image patch including line and point in low-resolution image data, matching feature pairs of point features between the image patch and previous image patch according to movement of the camera, and producing line of subpixel for the image patch, and estimating a location of the camera in the mobile devices based on difference between the produced line and estimated line by inertia.

First claim

The invention claimed is:

1. An operation method of a localization system for providing augmented reality in mobile devices, a localization method for augmented reality comprising:
sub-sampling first image data acquired from a camera in a mobile device, and extracting an image patch including line and point in the sub-sampled image data;
matching feature pairs of point features between the image patch and a previous image patch according to movement of the camera, and producing a line of subpixel for the image patch; and
estimating a location of the camera in the mobile devices based on difference between the produced line and estimated line by inertia.

2. The localization method of claim 1 , wherein the extracting image patch comprises:
receiving the first image data from the camera in the mobile device;
acquiring the sub-sampled image data by sub-sampling the first image data;
extracting features of line and point from the sub-sampled image data; and
extracting the image patch including the features of line and of point.

3. The localization method of claim 2 , wherein the extracting the image patch including the features of line and point extracts a circular image patch including all of line features and point features from the sub-sampled image data.

4. The localization method of claim 1 , wherein the matching feature pairs and producing line of subpixel for the image patch comprise:
confirming whether it is matched or not for feature pairs by using an IMU (Inertial Measurement Unit) and point features in the image patch; and
extracting the line of subpixel for the image patch by 3D line distribution.

5. The localization method of claim 4 , wherein the confirming whether it is matched or not for feature pairs searches point features of image patch (t) according to the movement of the camera [R/t] detected from the IMU on the basis of previous image patch (t−1), and acquires a plurality of feature pairs having the same straight line segment by matching each of the point features of the previous image patch (t−1) and the point features of the image patch (t).

6. The localization method of claim 5 , wherein the extracting the line of subpixel for the image patch expands 3D distribution for the image patch (t) by using the 3D line distribution of 3D Gaussian kernel, produces maximum line connecting maximum point in 3D distribution result, and acquires the accurate subpixel line.

7. The localization method of claim 1 , wherein the estimating a location of the camera in the mobile devices estimates the location of the camera by producing innovation term between the estimated line for the previous image patch (t−1) acquired by using an Inertial Navigation Sensor included in the mobile devices and the produced line for the image patch (t).

8. The localization method of claim 7 , wherein the estimating the location of the camera in the mobile devices improves accuracy of the estimate of the location of the camera (state x) by considering gradient difference between line gradient (at) of the produced line in the image patch (t) and line gradient (at−1) of the estimated line in the previous image patch (t−1).

9. A localization system for providing augmented reality in mobile devices, a localization system for augmented reality comprising:
an image patch extracting unit for sub-sampling first image data acquired from a camera in a mobile device, and extracting an image patch including line and point from the sub-sampled image data;
a producing unit for matching feature pairs of point features between the image patch and a previous image patch according to movement of the camera, and producing a line of subpixel for the image patch; and
an estimating unit for estimating a location of the camera in the mobile devices based on difference between the produced line and estimated line by inertia.

10. The localization system of claim 9 , wherein the image patch extracting unit comprises:
an image data receiving unit for receiving the first image data acquired from the camera in the mobile device;
a sub-sampled image data acquiring unit for acquiring the sub-sampled image data including line and point by sub-sampling the first image data; and
a feature extracting unit for extracting the image patch including features of line and point from the sub-sampled image data.

11. The localization system of claim 10 , wherein the feature extracting unit extracts a circular image patch including line features and point features from the sub-sampled image data.

12. The localization system of claim 9 , wherein the producing unit comprises:
a matching confirming unit for confirming whether it is matched or not for feature pairs by using an IMU (Inertial Measurement Unit) and point features in the image patch; and
a line extracting unit for extracting the line of subpixel for the image patch by 3D line distribution.

13. The localization system of claim 12 , wherein the matching confirming unit searches point features of image patch (t) according to the movement of the camera [R/t] detected from the IMU in the basis of previous image patch (t−1), and acquires a plurality of feature pairs having the same straight line segment by matching each of the point features of the previous image patch (t−1) and the point features of the image patch (t).

14. The localization system of claim 13 , wherein the line extracting unit expands 3D distribution for the image patch (t) by using the 3D line distribution of 3D Gaussian kernel, produces maximum line connecting maximum point in 3D distribution result, and extracts the accurate subpixel.

15. The localization system of claim 9 , wherein the location estimating unit estimates the location of the camera by producing innovation term between the estimated line for the previous image patch (t−1) acquired by using an Inertial Navigation Sensor included in the mobile devices and the produced line for the image patch (t).

16. The localization system of claim 15 , wherein the location estimating unit improves accuracy of the estimate of the location of the camera (state x) by considering gradient difference between line gradient (at) of the produced line in the image patch (t) and line gradient (at−1) of the estimated line in the previous image patch (t−1).

Assignees

• Korea Advanced Inst Sci & Tech

Inventors

• Myeong Hyeon
• Jung Kwang Yik
• Youn Pillip
• Kim Yeeun
• Lim Hyunjun
• Song Seungwon

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title