Method for representing virtual information in a real environment

Patent metadata

Field	Value
Publication number	US-11315308-B2
Application number	US-202017129340-A
Country	US
Kind code	B2
Filing date	Dec 21, 2020
Priority date	Dec 21, 2012
Publication date	Apr 26, 2022
Grant date	Apr 26, 2022

Abstract

A method for representing virtual information in a view of a real environment is provided that includes: providing a system setup including at least one display device, wherein the system setup is adapted for blending in virtual information on the display device in at least part of the view, determining a position and orientation of a viewing point relative to at least one component of the real environment, providing a geometry model of the real environment, providing at least one item of virtual information and a position of the at least one item of virtual information, determining whether the position of the item of virtual information is inside a 2D or 3D geometrical shape, determining a criterion which is indicative of whether the built-in real object is at least partially visible or non-visible in the view of the real environment, and blending in the at least one item of virtual information on the display device in at least part of the view of the real environment.

First claim

The invention claimed is:

1. A method for representing virtual information in a view of a real environment, comprising the following steps:
determining a position and orientation of a viewing point associated with a view of a real environment;
obtaining, based on the determined position and orientation of the viewing point, a geometry model having at least one boundary for delimiting at least one real object in the real environment;
obtaining an item of virtual information about a point of interest in the view of the real environment and position information for the point of interest;
calculating, based on the position information and the geometry model, a ray between the viewing point and the point of interest;
determining a number of boundary intersections by the ray across the at least one boundary for delimiting the at least one real object; and
determining a visualization mode for the item of virtual information based on the number of boundary intersections by the ray.

2. The method of claim 1 , wherein in response to the number of boundary intersections being less than two, the visualization mode for the item of virtual information is a non-occlusion mode.

3. The method according to claim 2 , wherein the non-occlusion mode corresponds to display of the item of virtual information as at least one of opaque display, in solid lines, or in a particular color associated with the non-occlusion mode.

4. The method of claim 1 , wherein in response to the number of boundary intersections being greater than or equal to two, the visualization mode for the item of virtual information is an occlusion mode.

5. The method according to claim 4 , wherein the occlusion mode corresponds to display of the item of virtual information as at least one of semi-transparent and in dash lines.

6. The method of claim 1 , wherein in response to the number of boundary intersections being greater than or equal to two, a display position for the item of virtual information is set to a front of an intersected boundary with respect to the position of the viewing point.

7. The method according to claim 1 , wherein:
in response to the number of boundary intersections being greater than or equal to two, the method is performed iteratively;
for each iteration, a height of a display position for the item of virtual information is increased; and
the iteration is stopped in response to a first of the number of boundary intersections being less than two and the height of the display position for the item of virtual information exceeding a predefined limit.

8. The method according to claim 1 , wherein determining the visualization mode is based at least in part on a distance from the point of interest to the viewing point, the method further comprising adding a graphical representation of the item of virtual information to the geometry model in response to the visualization mode being a non-occlusion mode.

9. The method according to claim 1 , wherein the geometry model is derived from a depth sensor source associated with the display device.

10. A non-transitory computer readable medium comprising computer instructions which, when executed by a computer system, cause the computer system to:
determine a position and orientation of a viewing point associated with a view of a real environment;
obtain, based on the determined position and orientation of the viewing point, a geometry model having at least one boundary for delimiting at least one real object in the real environment;
obtain an item of virtual information about a point of interest in the view of the real environment and position information for the point of interest;
calculate, based on the position information and the geometry model, a ray between the viewing point and the point of interest;
determine a number of boundary intersections by the ray across the at least one boundary for delimiting the at least one real object; and
determine a visualization mode for the item of virtual information based on the number of boundary intersections by the ray.

11. The non-transitory computer readable medium of claim 10 , wherein in response to the number of boundary intersections being less than two, the visualization mode for the item of virtual information is a non-occlusion mode.

12. The non-transitory computer readable medium of claim 11 , wherein:
the computer instructions to determine the visualization mode comprise computer instructions to determine the visualization mode based at least in part on a distance from the point of interest to the viewing point; and
the non-transitory computer readable medium further comprises computer instructions which, when executed by the computer system, cause the computer system to add a graphical representation of the item of virtual information to the geometry model in response to the visualization mode being the non-occlusion mode.

13. The non-transitory computer readable medium of claim 10 , wherein in response to the number of boundary intersections being greater than or equal to two, the visualization mode for the item of virtual information is an occlusion mode.

14. The non-transitory computer readable medium of claim 10 , wherein in response to the number of boundary intersections being greater than or equal to two, a display position for the item of virtual information is set to a front of an intersected boundary with respect to the position of the viewing point.

15. The non-transitory computer readable medium of claim 10 , further comprising computer instructions which, when executed by the computer system, cause the computer system to:
in response to the number of boundary intersections being greater than or equal to two, iteratively:
calculate, based on the position information and the geometry model, an additional ray between the viewing point and the point of interest; and
determine a number of boundary intersections by the additional ray, wherein for each iteration, a height of a display position for the item of virtual information is increased; and

stop the iteration in response to a first of the number of boundary intersections being less than two and the height of the display position for the item of virtual information exceeding a predefined limit.

16. A computing system, comprising:
a memory comprising computer instructions executable by the computing system to;
determine a position and orientation of a viewing point associated with a view of a real environment;
obtain, based on the determined position and orientation of the viewing point, a geometry model having at least one boundary for delimiting at least one real object in the real environment;
obtain an item of virtual information about a point of interest in the view of the real environment and position information for the point of interest;
calculate, based on the position information and the geometry model, a ray between the viewing point and point of interest;
determine a number of boundary intersections by the ray across the at least one boundary for delimiting the at least one real object; and
determine a visualization mode for the item of virtual information based on the number of boundary intersections by the ray.

17. The system of claim 16 , wherein in response to the number of boundary intersections being less than two, the visualization mode for the item of virtual information is a non-occlusion mode.

18. The system of claim 17 , wherein:
the computer instructions to determine the visualization mode comprise computer instructions to determine the visualization mode based at least in p

Assignees

• Apple Inc

Inventors

• Wang Lejing
• Meier Peter
• Misslinger Stefan

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T15/20 Primary

  Perspective computation · CPC title

• G06T15/06

  Ray-tracing · CPC title