Method for representing virtual information in a view of a real environment

Patent metadata

Field	Value
Publication number	US-11880951-B2
Application number	US-202217818255-A
Country	US
Kind code	B2
Filing date	Aug 8, 2022
Priority date	Oct 12, 2009
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

A method for representing virtual information in a view of a real environment comprises providing a virtual object having a global position and orientation with respect to a geographic global coordinate system, with first pose data on the global position and orientation of the virtual object, in a database of a server, taking an image of a real environment by a mobile device and providing second pose data as to at which position and with which orientation with respect to the geographic global coordinate system the image was taken. The method further includes displaying the image on a display of the mobile device, accessing the virtual object in the database and positioning the virtual object in the image on the basis of the first and second pose data, manipulating the virtual object or adding a further virtual object, and providing the manipulated virtual object with modified first pose data or the further virtual object with third pose data in the database.

First claim

The invention claimed is:

1. A method comprising:
capturing an image of an environment comprising a real object;
comparing a real object in the image to a corresponding real object of a reference image having an associated reference pose;
accessing a virtual object and a corresponding virtual object pose from a data store of a remote device for placement of the virtual object in a view of the environment captured by a local device;
determining an updated virtual object pose based on the reference pose and the comparing of the real object in the image to the corresponding real object in the reference image; and
updating the virtual object pose to the updated virtual object pose in the data store of the remote device the reference pose, wherein the virtual object is provided for display in accordance with the updated virtual object pose.

2. The method of claim 1 , further comprising:
sending a request to store the updated virtual object pose for the virtual object in the remote device.

3. The method of claim 1 , further comprising:
determining an image pose of the image of the environment in a reference coordinate system.

4. The method of claim 3 , wherein the reference coordinate system is determined based on global position system data.

5. The method of claim 1 , wherein the remote device comprises a reference database having reference views of the environment with reference pose data for the reference views.

6. The method of dais 5 , further comprising:
determining a position and orientation from which the image is captured in accordance with the matching.

7. The method of claim 1 , further comprising:
placing the virtual object in the image in accordance with the updated virtual object pose.

8. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
capture an image of an environment comprising a real object;
compare a real object in the image to a corresponding real object of a reference image having an associated reference pose;
access a virtual object and a corresponding virtual object pose from a data store of a remote device for placement of the virtual object in a view of the environment captured by a local device;
determining an updated virtual object pose based on the reference pose and the comparing of the real object in the image to the corresponding real object in the reference image; and
update the virtual object pose to the updated virtual object pose in the data store of the remote device the reference pose, wherein the virtual object is provided for display in accordance with the updated virtual object pose.

9. The non-transitory computer readable medium of claim 8 , further comprising computer readable code to:
send a request to store the updated virtual object pose for the virtual object in the remote device.

10. The non-transitory computer readable medium of claim 8 , further comprising computer readable code to:
determine an image pose of the image of the environment in a reference coordinate system.

11. The non-transitory computer readable medium of claim 10 , wherein the reference coordinate system is determined based on global position system data.

12. The non-transitory computer readable medium of claim 8 , wherein the remote device comprises a reference database having reference views of the environment with reference pose data for the reference views.

13. The non-transitory computer readable medium of claim 12 , further comprising computer readable code to:
determine a position and orientation from which the image is captured in accordance with the matching.

14. The non-transitory computer readable medium of claim 12 , further comprising computer readable rode to:
place the virtual object in the image in accordance with the updated virtual object pose.

15. A system comprising:
one or more processors; and
one or more computer readable media comprising computer readable code executable by the one or more processors to:
capture an image of an environment comprising a real object;
compare a real object in the image to a corresponding real object of a reference image having an associated reference pose;
access a virtual object and a corresponding virtual object pose from a data store of a remote device for placement of the virtual object in a view of the environment captured by a local device;
determining an updated virtual object pose based on the reference pose and the comparing of the real object in the image to the corresponding real object in the reference image; and
update the virtual object pose to the updated virtual object pose in the data store of the remote device the reference pose, wherein the virtual object is provided for display in accordance with the updated virtual object pose.

16. The system of claim 15 , further comprising computer readable code to:
send a request to store the updated virtual object pose for the virtual object in the remote device.

17. The system of claim 15 , further comprising computer readable code to:
determine an image pose of the image of the environment in a reference coordinate system based on global position system data.

18. The system of claim 15 , wherein the remote device comprises a reference database having reference views of the environment with reference pose data for the reference views.

19. The system of claim 15 , further comprising computer readable code to:
determine a position and orientation from which the image is captured in accordance with the matching.

20. The system of claim 19 , further comprising computer readable code to:
place the virtual object in the image in accordance with the updated virtual object pose.

Assignees

• Apple Inc

Inventors

• Meier Peter
• Kuhn Michael
• Angermann Frank

Classifications

• G01S19/01

  Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T2219/024

  Multi-user, collaborative environment · CPC title