Online reference generation and tracking for multi-user augmented reality

Patent metadata

Field	Value
Publication number	US-9558557-B2
Application number	US-201514662555-A
Country	US
Kind code	B2
Filing date	Mar 19, 2015
Priority date	Sep 9, 2010
Publication date	Jan 31, 2017
Grant date	Jan 31, 2017

Abstract

A multi-user augmented reality (AR) system operates without a previously acquired common reference by generating a reference image on the fly. The reference image is produced by capturing at least two images of a planar object and using the images to determine a pose (position and orientation) of a first mobile platform with respect to the planar object. Based on the orientation of the mobile platform, an image of the planar object, which may be one of the initial images or a subsequently captured image, is warped to produce the reference image of a front view of the planar object. The reference image may be produced by the mobile platform or by, e.g., a server. Other mobile platforms may determine their pose with respect to the planar object using the reference image to perform a multi-user augmented reality application.

First claim

What is claimed is:

1. A method comprising:
receiving a first image of a planar object captured by a first mobile platform from the first mobile platform;
receiving a second image of the planar object captured by a second mobile platform from the second mobile platform;
determining a first pose, comprising a position and an orientation, of the first mobile platform with respect to the planar object using the first and second images;
warping the first image of the planar object using the orientation of the first mobile platform with respect to the planar object to produce a reference image of a front view of the planar object; and
transmitting the reference image to the first and second platforms, wherein the second platform determines a second pose of the second platform with respect to the planar object based at least in part on the reference image and the second image captured by the second platform, wherein determining the first pose, comprising the position and the orientation, of the first mobile platform with respect to the planar object comprises:
extracting features from the first and second images;
computing a 2D-2D homography between the first and second images using the features;
computing 3D coordinates for the features to define a 3D plane that corresponds to a plane of the planar object;
computing a 3D-2D homography between the 3D plane and one of the first and second images using the features and the 3D plane; and
using the 3D-2D homography to determine the first pose.

2. The method of claim 1 , further comprising performing an augmented reality application with the second mobile platform based on the second pose of the second mobile platform with respect to the planar object.

3. The method of claim 1 , wherein the second platform determines the second pose of the second platform based at least in part on the reference image and the second image of the planar object captured by the second platform by performing a method comprising:
extracting features from the reference image;
defining 3D points on a 3D plane that correspond to the features extracted from the reference image;
finding points in the second image of the planar object captured by the second mobile platform that match with 2D projections of the 3D points; and
using matched 3D-2D point pairs to estimate the second pose of the second mobile platform.

4. The method of claim 3 , wherein the features extracted from the reference image comprise point features or line features.

5. The method of claim 1 , further comprising:
receiving a new image captured by the second mobile platform from the second mobile platform that includes a portion of the planar object that is not included in the reference image of the front view of the planar object; and
merging the new image and the reference image to generate a new reference image of the front view of the planar object; and
transmitting the new reference image to the first and second mobile platforms, wherein the second mobile platform determines an updated second pose of the second mobile platform with respect to the planar object using a subsequent image of the planar object captured by the second mobile platform and the new reference image.

6. The method of claim 5 , wherein merging the new image and the reference image to generate the new reference image of the front view of the planar object comprises transforming pixels of the reference image to the new reference image through an identity transformation and transforming pixels of the new image to the new reference image.

7. The method of claim 6 , wherein transforming pixels of the new image to the new reference image is performed using at least one of a computed homography or affine transformation.

8. An apparatus comprising:
means for receiving a first image of a planar object captured by a first mobile platform from the first mobile platform;
means for receiving a second image of the planar object captured by a second mobile platform from the second mobile platform;
means for determining a first pose, comprising a position and an orientation, of the first mobile platform with respect to the planar object using the first and second images;
means for warping the first image of the planar object using the orientation of the first mobile platform with respect to the planar object to produce a reference image of a front view of the planar object; and
means for transmitting the reference image to the first and second platforms, wherein the second platform determines a second pose of the second platform with respect to the planar object based at least in part on the reference image and the second image captured by the second platform, wherein the means for determining the first pose, comprising the position and the orientation, of the first mobile platform with respect to the planar object comprises:
means for extracting features from the first and second images;
means for computing a 2D-2D homography between the first and second images using the features;
means for computing 3D coordinates for the features to define a 3D plane that corresponds to a plane of the planar object;
means for computing a 3D-2D homography between the 3D plane and one of the first and second images using the features and the 3D plane; and
means for using the 3D-2D homography to determine the first pose.

9. The apparatus of claim 8 , further comprising means for performing an augmented reality application with the second mobile platform based on the second pose of the second mobile platform with respect to the planar object.

10. The apparatus of claim 8 , wherein the second platform determines the second pose of the second platform based at least in part on the reference image and the second image of the planar object captured by the second platform by performing a method comprising:
extracting features from the reference image;
defining 3D points on a 3D plane that correspond to the features extracted from the reference image;
finding points in the second image of the planar object captured by the second mobile platform that match with 2D projections of the 3D points; and
using matched 3D-2D point pairs to estimate the second pose of the second mobile platform.

11. The apparatus of claim 10 , wherein the features extracted from the reference image comprise point features or line features.

12. The apparatus of claim 8 , further comprising:
means for receiving a new image captured by the second mobile platform from the second mobile platform that includes a portion of the planar object that is not included in the reference image of the front view of the planar object; and
means for merging the new image and the reference image to generate a new reference image of the front view of the planar object; and
means for transmitting the new reference image to the first and second mobile platforms, wherein the second mobile platform determines an updated second pose of the second mobile platform with respect to the planar object using a subsequent image of the planar object captured by the second mobile platform and the new reference image.

13. The apparatus of claim 12 , wherein the means for merging the new image and the reference image to generate the new reference image of the front view of the planar object comprises means for transforming pixels of the reference image to the new reference image through an identity transformation and means for transforming pixels of the new image to the new reference image.

14. The apparatus of claim 13 , wherein transforming pixels of the new image to the new reference image is performed using at least one of a computed homog

Assignees

• Qualcomm Inc

Inventors

• Jiang Bolan
• Swaminathan Ashwin
• Spindola Serafin Diaz
• Ahuja Dheeraj
• Sweet Iii Charles Wheeler
• Choudhury Prasun

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T7/0044 Primary

  Physics · mapped topic

• G06T7/00 Primary

  Image analysis · CPC title