Method for establishing a common reference frame amongst devices for an augmented reality session

Patent metadata

Field	Value
Publication number	US-11798237-B2
Application number	US-202117229465-A
Country	US
Kind code	B2
Filing date	Apr 13, 2021
Priority date	May 31, 2018
Publication date	Oct 24, 2023
Grant date	Oct 24, 2023

Abstract

A method includes: detecting an object in a first image; receiving a selection of the object depicted in the image; associating the object with a second device based on the selection; and, in response to the selection: recording a series of odometry data; estimating a location of the first device based on the odometry data; recording a series of images; estimating a location of the second device based on the images; calculating a first reference vector in the reference frame of the first device defining the location of the second device relative to the location of the first device; receiving, from the second device, a second reference vector; calculating a rotation and an offset between the reference vectors; and transforming the reference frame of the first device to a common reference based on the rotation and the offset.

First claim

We claim:

1. A method for establishing a common reference frame amongst devices comprising, at a first device:
recording a first image;
detecting a first object in the first image;
rendering the first image;
receiving a selection of the first object depicted in the first image;
associating the first object with a second device based on the selection of the first object; and
in response to the selection:
recording a first series of odometry data;
at a first coordination time, estimating a first location of the first device in a reference frame of the first device based on the first series of odometry data;
recording a first series of images;
at the first coordination time, estimating a second location of the second device in the reference frame of the first device based on the first series of images;
calculating a first reference vector in the reference frame of the first device defining the second location of the second device relative to the first location of the first device;
receiving, from the second device, a second reference vector in the reference frame of the second device defining the first location of the first device relative to the second location of the second device;
calculating a rotation and an offset between the second reference vector and the first reference vector; and
transforming the reference frame of the first device to the common reference frame matching the reference frame of the second device based on the rotation and the offset.

2. The method of claim 1 further comprising, at the second device:
recording a second image;
detecting a second object in the second image;
rendering the second image;
receiving a selection of the second object depicted in the second image;
associating the second object with the first device based on the selection of the second object; and
in response to the selection:
recording a second series of odometry data;
at a coordination time, estimating a second location of the second device in a reference frame of the second device based on the second series of odometry data;
recording a second series of images;
at the coordination time, estimating the first location of the first device in the reference frame of the second device based on the second series of images; and
calculating a second reference vector in the reference frame of the second device defining the first location of the first device relative to the second location of the second device; and
transmitting, from the second device, the second reference vector in the reference frame of the second device.

3. The method of claim 1 , wherein:
detecting the first object in the first image further comprises detecting a first human in the first image;
receiving the selection of the first object depicted in the first image further comprises receiving a selection of the first human in the first image;
associating the first object with the second device based on the selection of the first object further comprises identifying the first human as a user of the second device based on the selection of the first human; and
estimating the second location of the second device in the reference frame of the first device based on the first series of images further comprises:
estimating a centroid location of the user of the second device in the reference frame of the first device based on the first series of images; and
estimating the second location of the second device in the reference frame of the first device based on the centroid location of the user of the second device and a typical offset between a device and a centroid of a user.

4. The method of claim 1 :
further comprising augmenting the first image in real time to graphically identify the first object in the first image to generate a first augmented image; and
wherein rendering the first image further comprises rendering the first augmented image; and
wherein receiving the selection of the first object depicted in the first image further comprises detecting a touch gesture over the first object depicted in the first image displayed on a touchscreen of the first device.

5. The method of claim 4 , wherein:
recording the first image further comprises recording a first video feed comprising the first image;
detecting the first object in the first image further comprises tracking the first object in the first video feed;
augmenting the first image in real time to graphically identify the first object in the first image to generate the first augmented image further comprises augmenting the first video feed in real time to graphically identify the first object in the first video feed to generate a first augmented video feed;
rendering the first augmented image further comprises rendering the first augmented video feed in real time; and
receiving the selection of the first object depicted in the first image further comprises receiving the selection of the first object depicted in the first video feed.

6. The method of claim 1 , further comprising:
recording a second series of odometry data;
localizing the first device based on the second series of odometry data at a third location in the common reference frame;
recording a second series of images comprising a second image;
detecting a third object in the second image;
rendering the second image;
receiving a selection of the third object depicted in the second image;
associating the third object with the second device based on the selection of the first object;
at a second coordination time subsequent to the first coordination time:
estimating a fourth location of the second device in the reference frame of the first device based on the second series of images; and
calculating a third reference vector in the reference frame of the first device defining the fourth location of the second device relative to the third location of the first device;

receiving, from the second device, a fourth reference vector in the reference frame of the second device defining the third location of the first device relative to the fourth location of the second device;
calculating a second rotation and a second offset between the third reference vector and the fourth reference vector; and
updating the common reference frame for the first device and the second device based on the second rotation and the second offset.

7. The method of claim 1 further comprising:
calculating a unique object signature associated with the first object based on the first image;
recording a second series of odometry data;
localizing the first device based on the second series of odometry data at a third location in the common reference frame recording a second series of images comprising a second image;
at a second coordination time subsequent to the first coordination time:
detecting an object in the second image;
calculating an object signature for the object in the second image based on the second image;
in response to calculating substantial similarity between the unique object signature and the object signature for the object in the second image:
associating the object in the second image with the second device;
estimating a fourth location of the second device in the reference frame of the first device based on the second series of images; and
calculating a third reference vector in the reference frame of the first device defining the fourth location of the second device relative to the third location of the first device;

receiving, from the second device, a fourth reference vector in the reference frame of the second device defining the third location of the first device relative to the fourth location of the second device;
calculating a second rotation and a second offset between the third reference vector and the fourth

Assignees

• Roblox Corp

Inventors

• Stauber Mark
• Sung Jaeyong
• Levy Amichai

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2210/12

  Bounding box · CPC title

• G06T2219/024

  Multi-user, collaborative environment · CPC title