Method for representing virtual information in a view of a real environment

Patent metadata

Field	Value
Publication number	US-12499636-B2
Application number	US-202318538656-A
Country	US
Kind code	B2
Filing date	Dec 13, 2023
Priority date	Oct 12, 2009
Publication date	Dec 16, 2025
Grant date	Dec 16, 2025

Abstract

A method for representing virtual information in a view of a real environment comprises providing a virtual object having a global position and orientation with respect to a geographic global coordinate system, with first pose data on the global position and orientation of the virtual object, in a database of a server, taking an image of a real environment by a mobile device and providing second pose data as to at which position and with which orientation with respect to the geographic global coordinate system the image was taken. The method further includes displaying the image on a display of the mobile device, accessing the virtual object in the database and positioning the virtual object in the image on the basis of the first and second pose data, manipulating the virtual object or adding a further virtual object, and providing the manipulated virtual object with modified first pose data or the further virtual object with third pose data in the database.

First claim

The invention claimed is:

1 . A method comprising:
receiving an image from a computing device associated with an initial image pose and comprising a virtual object positioned in the image; matching features of a real object depicted in the image to corresponding features of a real object depicted in a reference image having a reference image pose; generating an updated image pose based at least in part on the reference image pose in response to the feature matching; determining an updated global object pose for the virtual object based on the updated image pose; and providing the virtual object and the updated global object pose to a second computing device.

2 . The method of claim 1 , wherein the initial image pose is determined using a first localization technique, and wherein the reference image pose is determined using a second localization technique.

3 . The method of claim 1 , further comprising:
storing the virtual object with the updated global object pose in server storage.

4 . The method of claim 1 , wherein the virtual object is presented on a display of the second computing device in accordance with the updated global object pose.

5 . The method of claim 1 , further comprising:
receiving an indication of a manipulated object pose from the virtual object; and storing the manipulated object pose with the virtual object.

6 . The method of claim 1 , wherein the virtual object is presented from a first perspective when presented in the image, and wherein the virtual object is presented from a second perspective when presented at the second computing device.

7 . A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
receive an image from a computing device associated with an initial image pose and comprising a virtual object positioned in the image; match features of a real object depicted in the image to corresponding features of a real object depicted in a reference image having a reference image pose; generate an updated image pose based at least in part on the reference image pose in response to the feature matching; determine an updated global object pose for the virtual object based on the updated image pose; and provide the virtual object and the updated global object pose to a second computing device.

8 . The non-transitory computer readable medium of claim 7 , wherein the initial image pose is determined using a first localization technique, and wherein the reference image pose is determined using a second localization technique.

9 . The non-transitory computer readable medium of claim 7 , further comprising computer readable code to:
store the virtual object with the updated global object pose in server storage.

10 . The non-transitory computer readable medium of claim 8 , wherein the virtual object is presented on a display of the second computing device in accordance with the updated global object pose.

11 . The non-transitory computer readable medium of claim 7 , further comprising computer readable code to:
receive an indication of a manipulated object pose from the virtual object; and store the manipulated object pose with the virtual object.

12 . The non-transitory computer readable medium of claim 7 , wherein the virtual object is presented from a first perspective when presented in the image, and wherein the virtual object is presented from a second perspective when presented at the second computing device.

13 . A system comprising:
one or more processors; and one or more computer readable media comprising computer readable code executable by the one or more processors to:
receive an image from a computing device associated with an initial image pose and comprising a virtual object positioned in the image;
match features of a real object depicted in the image to corresponding features of a real object depicted in a reference image having a reference image pose;
generate an updated image pose based at least in part on the reference image pose in response to the feature matching;
determine an updated global object pose for the virtual object based on the updated image pose; and
provide the virtual object and the updated global object pose to a second computing device.

14 . The system of claim 13 , further comprising computer readable code to:
store the virtual object with the updated object pose in server storage.

15 . The system of claim 13 , wherein the virtual object is presented on a display of the second computing device in accordance with the updated global object pose.

16 . The system of claim 13 , further comprising computer readable code to:
receive an indication of a manipulated object pose from the virtual object; and store the manipulated object pose with the virtual object.

17 . The system of claim 13 , wherein the virtual object is presented from a first perspective when presented in the image, and wherein the virtual object is presented from a second perspective when presented at the second computing device.

Assignees

• Apple Inc

Inventors

• Meier Peter
• Kuhn Michael
• Angermann Frank

Classifications

• G06T2219/024

  Multi-user, collaborative environment · CPC title

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G01S19/01

  Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title