Methods for calibrating augmented reality scenes

Patent metadata

Field	Value
Publication number	US-12423924-B2
Application number	US-202218082706-A
Country	US
Kind code	B2
Filing date	Dec 16, 2022
Priority date	Sep 9, 2022
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

A computer-implemented method is disclosed. The method includes: determining a first position of a real display device in a real-world environment; receiving a request to display virtual information at a second relative position with respect to the first position in an AR version of the real-world environment; responsive to receiving the request: determining a displayable area associated with the second relative position; and causing the virtual information to be overlaid on the displayable area in the AR version of the real-world environment.

First claim

The invention claimed is:

1. A computer-implemented method, comprising:
determining a first position of a real display device in a real-world environment;
selectively causing to be displayed, on a displayable area associated with the real display device, a software-generated virtual visual marker for use in positional synchronization of an AR scene, wherein the virtual visual marker encodes a unique identifier of the real display device and at least one of location, orientation, or scale of the virtual visual marker relative to the real display device and wherein the virtual visual marker is caused to be displayed in response to detecting a calibration drift;
receiving a request to display virtual information on a virtual display at a second relative position with respect to the first position in an AR version of the real-world environment;
responsive to receiving the request:
determining a displayable area of the virtual display in the AR scene, wherein determining the displayable area comprises:
detecting the virtual visual marker in image data depicting the real display device; and
causing the displayable area of the virtual display to be anchored on the real display device based on the detected virtual visual marker; and

causing the virtual information to be overlaid on the displayable area in the AR version of the real-world environment.

2. The method of claim 1 , wherein determining the first position comprises determining real-world space coordinates associated with a location of the real display device in the real-world environment.

3. The method of claim 1 , further comprising monitoring user interaction with the real display device, wherein the request to display the virtual information comprises a detected user interaction input associated with the real display device.

4. The method of claim 3 , wherein the user interaction input comprises one of: a drag-and-drop gesture using an input device; a gesture for moving one or more UI elements displayed on a display area of the real display device; or a gesture for moving one or more virtual UI elements shown as overlay on the real display device in AR.

5. The method of claim 3 , wherein the second relative position is determined based on an end position associated with the detected user interaction input.

6. The method of claim 1 , further comprising obtaining sensor data of sensors for tracking gestures of the user, wherein gestures of the user in the real-world environment are detected based on the obtained sensor data.

7. The method of claim 6 , wherein the sensors comprise at least one of: cameras; LiDAR array; eye trackers; or hand trackers.

8. The method of claim 1 , wherein the virtual visual marker comprises at least one of a pattern or a fiducial.

9. The method of claim 1 , further comprising:
determining that a positional synchronization with the real display device has been lost;
responsive to determining that the positional synchronization has been lost:
obtaining image data captured using cameras associated with an AR-enabled computing device;
detecting the virtual visual marker in the image data; and
causing the AR scene to be positionally synchronized based on the detected virtual visual marker.

10. The method of claim 1 , further comprising:
determining that a positional synchronization with the real display device has been lost;
responsive to determining that the positional synchronization has been lost:
determining a defined third position in the real-world environment; and
causing the AR scene to be positionally synchronized relative to the defined third position.

11. The method of claim 10 , wherein the defined third position comprises one of: a last stored position of the real display device; a location of a detectable landmark in the AR version of the real-world environment; or a current position of an AR-enabled computing device.

12. The method of claim 1 , wherein the displayable area of the virtual display comprises a virtual display screen overlaid on a view of the real-world environment.

13. The method of claim 1 , wherein causing the virtual visual marker to be displayed comprises:
detecting a trigger condition for changing visibility of the virtual visual marker;
in response to detecting the trigger condition, generating display data for the virtual visual marker; and
causing the display data to be rendered on the real display device.

14. A computing system, comprising:
a processor;
a memory coupled to the processor, the memory storing computer-executable instructions that, when executed by the processor, configure the processor to:
determine a first position of a real display device in a real-world environment;
selectively cause to be displayed, on a displayable area associated with the real display device, a software-generated virtual visual marker for use in positional synchronization of an AR scene, wherein the virtual visual marker encodes a unique identifier of the real display device and at least one of location, orientation, or scale of the virtual visual marker relative to the real display device and wherein the virtual visual marker is caused to be displayed in response to detecting a calibration drift;
receive a request to display virtual information at a second relative position with respect to the first position in an AR version of the real-world environment;
responsive to receiving the request:
determine a displayable area of the virtual display in the AR scene, wherein determining the displayable area comprises:
detecting the virtual visual marker in image data depicting the real display device; and
causing the displayable area of the virtual display to be anchored on the real display device in the AR scene based on the detected virtual visual marker; and

cause the virtual information to be overlaid on the displayable area in the AR version of the real-world environment.

15. The computing system of claim 14 , wherein determining the first position comprises determining real-world space coordinates associated with a location of the real display device in the real-world environment.

16. The computing system of claim 14 , wherein the instructions, when executed by the processor, further configure the processor to monitor user interaction with the real display device, wherein the request to display the virtual information comprises a detected user interaction input associated with the real display device.

17. The computing system of claim 16 , wherein the user interaction input comprises one of: a drag-and-drop gesture using an input device; a gesture for moving one or more UI elements displayed on a display area of the real display device; or a gesture for moving one or more virtual UI elements shown as overlay on the real display device in AR.

18. The computing system of claim 16 , wherein the second relative position is determined based on an end position associated with the detected user interaction input.

19. The computing system of claim 14 , wherein the instructions, when executed by the processor, further configure the processor to obtain sensor data of sensors for tracking gestures of the user, wherein gestures of the user in the real-world environment are detected based on the obtained sensor data.

20. A non-transitory, computer-readable medium storing computer-executable instructions that, when executed by a processor, configure the processor to:
determine a first position of a real display device in a real-world environment;
selectively cause to be displayed, on a

Assignees

• Shopify Inc

Inventors

• Padgett Neil Leonard
• Maschmeyer Russ
• Florenzano Eric Andrew
• Letkeman Brennan
• Ngo Andrew Nhat-Nam
• Jayatunga Hettige Ray Perera
• Lepp James
• Bello Diego Macario

Classifications

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06T3/14

  Transformations for image registration, e.g. adjusting or mapping for alignment of images · CPC title

• G06F3/0486 Primary

  Drag-and-drop · CPC title

• G06T2207/30204

  Marker · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title