Presenting labels in augmented reality

Patent metadata

Field	Value
Publication number	US-12039672-B2
Application number	US-202217828878-A
Country	US
Kind code	B2
Filing date	May 31, 2022
Priority date	Jun 6, 2021
Publication date	Jul 16, 2024
Grant date	Jul 16, 2024

Abstract

In some implementations, a computing device can present augmented reality (AR) labels in an AR video stream. For example, the computing device can obtain route information for a route requested by a user and can determine locations along the route for placing candidate AR labels. The computing device can determine the precise location of the computing device using camera depth information obtained in response to the user scanning the local real-world environment with a camera of the computing device. The computing device can select an AR label and/or label placement location for presentation in an AR video stream based on various criteria, including the distance between the candidate AR labels and the precise location of the computing device, priorities assigned to each candidate AR label, and/or whether a clear line of sight exists between the precise location of the computing device and the candidate AR label location.

First claim

What is claimed is:

1. A method comprising:
presenting, by a computing device, an augmented reality (AR) video stream on a display of the computing device, the augmented reality video stream presenting images captured by a camera of the computing device;
determining, by the computing device, a current location of the computing device;
determining, by the computing device, a plurality of candidate AR labels to present in the AR video stream based on the current location of the computing device;
selecting, by the computing device, a first AR label of the candidate AR labels based on a plurality of selection criteria, the selection criteria including a distance from a particular location associated with the first AR label and the current location of the computing device; and
presenting, by the computing device, the first AR label in the AR video stream on the display of the computing device.

2. The method of claim 1 , wherein the selection criteria further include at least one priority associated with the first AR label, the method further comprising:
determining a respective priority for each of the candidate AR labels; and
determining that the first AR label is a highest priority label in the plurality of candidate AR labels; and
selecting the first AR label when the first AR label is a highest priority label in the plurality of candidate AR labels.

3. The method of claim 2 , wherein the selection criteria includes a distance between the current location of the computing device and the candidate AR labels, further comprising:
determining that a plurality of highest priority AR labels in the plurality of candidate AR labels have the highest priority; and
determining a respective location for each of the plurality highest priority AR labels;
determining the distance between each respective location and the current location of the computing device; and
selecting the first AR label based on the distance between each respective location for each of the plurality highest priority AR labels and the current location of the computing device.

4. The method of claim 1 , wherein the selection criteria includes whether the computing device has a clear line of sight to the first AR label, and further comprising:
determining that a clear line of sight exists between the current location of the computing device and the respective location of the first AR label; and
selecting the first AR label when the clear line of sight exists.

5. The method of claim 4 , further comprising:
obtaining a three-dimensional (3D) mesh model of physical structures near the current location of the computing device;
extending rays from the current location of the computing device to a plurality of points on the first AR label;
determining whether each of the rays intersect a structural surface defined by the three-dimensional (3D) mesh model; and
selecting the first AR label when each of the rays fails to intersect the structural surface defined by the 3D mesh model.

6. The method of claim 1 , further comprising:
obtaining a three-dimensional (3D) mesh model of physical structures near the current location of the computing device;
extending rays from the current location of the computing device to a plurality of points on a second AR label in the plurality of candidate AR labels;
determining whether each of the rays intersect a structural surface defined by the three-dimensional (3D) mesh model; and
removing the second AR label from the plurality of candidate AR labels when at least one ray intersects the structural surface defined by the 3D mesh model, wherein removing the second AR label from the plurality of candidate AR labels prevents the presentation of the second AR label in the AR video stream on the display of the computing device.

7. The method of claim 4 , further comprising:
determining intermediate candidate AR labels having locations between the current location of the computing device and the respective location of the particular AR label;
obtaining, by the computing device, respective elevations associated with each respective location of the intermediate candidate AR labels and the particular AR label;
determining, by the computing device, a current elevation associated with the current location of the computing device;
determining whether the intermediate candidate AR labels are associated with an elevation that is higher than the current elevation of the computing device and the respective elevation of the particular AR label;
selecting the particular AR label when none of the intermediate candidate AR labels are associated with an elevation that is higher than the current elevation of the computing device and the respective elevation of the particular AR label.

8. A non-transitory computer readable medium including one or more sequences of instructions that, when executed by one or more processors, cause the processors to perform operations comprising:
presenting, by a computing device, an augmented reality (AR) video stream on a display of the computing device, the augmented reality video stream presenting images captured by a camera of the computing device;
determining, by the computing device, a current location of the computing device;
determining, by the computing device, a plurality of candidate AR labels to present in the AR video stream based on the current location of the computing device;
selecting, by the computing device, a first AR label of the candidate AR labels based on a plurality of selection criteria, the selection criteria including a distance from a particular location associated with the first AR label and the current location of the computing device; and
presenting, by the computing device, the first AR label in the AR video stream on the display of the computing device.

9. The non-transitory computer readable medium of claim 8 , wherein the selection criteria further include at least one priority associated with the first AR label, and wherein the instructions cause the processors to perform operations comprising:
determining a respective priority for each of the candidate AR labels; and
determining that the first AR label is a highest priority label in the plurality of candidate AR labels; and
selecting the first AR label when the first AR label is a highest priority label in the plurality of candidate AR labels.

10. The non-transitory computer readable medium of claim 9 , wherein the selection criteria includes a distance between the current location of the computing device and the candidate AR labels, and wherein the instructions cause the processors to perform operations comprising:
determining that a plurality of highest priority AR labels in the plurality of candidate AR labels have the highest priority; and
determining a respective location for each of the plurality highest priority AR labels;
determining the distance between each respective location and the current location of the computing device; and
selecting the first AR label based on the distance between each respective location for each of the plurality highest priority AR labels and the current location of the computing device.

11. The non-transitory computer readable medium of claim 8 , wherein the selection criteria includes whether the computing device has a clear line of sight to the first AR label, and wherein the instructions cause the processors to perform operations comprising:
determining that a clear line of sight exists between the current location of the computing device and the respective location of the first AR label; and
selecting the first AR label when the clear line of sight exists.

12. The non-transitory computer readable medium

Assignees

• Apple Inc

Inventors

• Wu ting-yuan
• Pasek Lukasz J
• Bhutani Ishan
• Hasan Syed Mohsin
• Uzum Vella Isil
• Sturm Eugene P
• Bangu Razvan
• Ahrens Paul F
• Ball Mathew B
• Coleman Patrick J
• Dreyer Benjamin R
• West Roy E
• Andrich Brian J
• Magharious George

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T15/06

  Ray-tracing · CPC title

• G06T17/05

  Geographic models · CPC title

• G06T2219/004

  Annotating, labelling · CPC title