Methods, systems, and articles of manufacture for augmented reality application for display of virtual objects

Patent metadata

Field	Value
Publication number	US-12266010-B2
Application number	US-202318470818-A
Country	US
Kind code	B2
Filing date	Sep 20, 2023
Priority date	Dec 21, 2018
Publication date	Apr 1, 2025
Grant date	Apr 1, 2025

Abstract

Disclosed are systems and methods for augmenting a customer image with at least one virtual object. One or more customer images depicting a customer environment are received from a customer computing device and 3D feature data for the customer images is determined. A virtual object is determined which corresponds to a desired merchant item. A positioning signal is received which corresponds to a desired location of virtual object in the customer environment, and a corresponding first location in each customer image at which to overlay the virtual object using the positioning signal is determined. An appropriate size and orientation of the virtual object is determined for each customer image based on corresponding 3D feature data, causing an overlay of an appropriately sized and oriented virtual object at the corresponding first location in customer images.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving a plurality of images from a computing device, the plurality of images of a real-world environment, wherein each of the plurality of images depicts a respective different 2D view of the real-world environment such that different images depict different points of view of the real-world environment;
determining 3D feature data corresponding to each of the plurality of images;
receiving a selection of an item for display;
determining a virtual object corresponding to the item;
receiving a positioning signal corresponding to a desired location of the virtual object in a particular 2D view of the real-world environment;
replicating and simultaneously displaying the virtual object overlaid on the real-world environment in each of the plurality of images depicting the respective different 2D view to result in simultaneous augmented reality (AR) views of the virtual object from the different points of view by:
determining a corresponding location in each of the plurality of images at which to overlay the virtual object based on the positioning signal, the corresponding location corresponding to the desired location in each respective different 2D view of the real-world environment;
determining a size and orientation of the virtual object in each of the plurality of images based on the corresponding 3D feature data; and
causing an overlay of the virtual object on the real-world environment, having the determined size and orientation, at the corresponding location in each of the plurality of images to create a plurality of augmented images for display; and

causing simultaneous display of the AR views of the virtual object from the different points of view with the virtual object overlaid on the real-world environment at different views by causing simultaneous display of each of the augmented images having the virtual object overlaid on the real-world environment.

2. The computer-implemented method of claim 1 , wherein the computing device is a first computing device, and wherein the selection of the item for display is received from a second computing device that is different from the first computing device.

3. The computer-implemented method of claim 1 , wherein the computing device is a first computing device, and wherein the positioning signal is received from a second computing device that is different from the first computing device.

4. The computer-implemented method of claim 1 , wherein the computing device is a first computing device, and wherein the plurality of augmented images are transmitted to a second computing device for simultaneous display, wherein the second computing device is different from the first computing device.

5. The computer-implemented method of claim 4 , wherein the first computing device is associated with a customer and the second computing device is associated with a merchant.

6. The computer-implemented method of claim 1 , wherein the selection of the item for display and the positioning signal are received from the computing device.

7. The computer-implemented method of claim 1 , wherein the plurality of augmented images are transmitted to the computing device for simultaneous display.

8. The computer-implemented method of claim 1 , wherein the corresponding location is a corresponding first location, and wherein the method further comprises:
receiving a re-positioning signal indicative of a desired re-positioned location of the virtual object in the particular 2D view or in another 2D view of the real-world environment;
determining a corresponding second location in each of the plurality of images at which to overlay the virtual object based on the re-positioning signal, the corresponding second location corresponding to the desired re-positioned location in each respective different 2D view of the real-world environment; and
updating the overlay of the virtual object to be positioned at the corresponding second location in each of the plurality of images.

9. The computer-implemented method of claim 1 , wherein the 3D feature data includes at least one of: camera position, camera orientation, position of a detected surface, position of a detected object, position of a detected feature, size and orientation of the detected surface, size and orientation of the detected object, or size and orientation of the detected feature.

10. The computer-implemented method of claim 1 , wherein the 3D feature data includes a determined coordinate system relative to a camera of the computing device.

11. A system comprising:
a processor;
a memory coupled to the processor, the memory storing instructions which, when executed by the processor, cause the system to:
receive a plurality of images from a computing device, the plurality of images of a real-world environment, wherein each of the plurality of images depicts a respective different 2D view of the real-world environment such that different images depict different points of view of the real-world environment;
determine 3D feature data corresponding to each of the plurality of images;
receive a selection of an item for display;
determine a virtual object corresponding to the item;
receive a positioning signal corresponding to a desired location of the virtual object in a particular 2D view of the real-world environment;
replicate and simultaneously display the virtual object overlaid on the real-world environment in each of the plurality of images depicting the respective different 2D view
to result in simultaneous augmented reality (AR) views of the virtual object from the different points of view by:
determining a corresponding location in each of the plurality of images at which to overlay the virtual object based on the positioning signal, the corresponding location corresponding to the desired location in each respective different 2D view of the real-world environment;
determining a size and orientation of the virtual object in each of the plurality of images based on the corresponding 3D feature data; and
causing an overlay of the virtual object on the real-world environment, having the determined size and orientation, at the corresponding location in each of the plurality of images to create a plurality of augmented images for display; and

cause simultaneous display of the AR views of the virtual object from the different points of view with the virtual object overlaid on the real-world environment at different views by causing simultaneous display of each of the augmented images having the virtual object overlaid on the real-world environment.

12. The system of claim 11 , wherein the computing device is a first computing device, and wherein the selection of the item for display is received from a second computing device that is different from the first computing device.

13. The system of claim 11 , wherein the computing device is a first computing device, and wherein the positioning signal is received from a second computing device that is different from the first computing device.

14. The system of claim 11 , wherein the computing device is a first computing device, and wherein the plurality of augmented images are for transmission to a second computing device for simultaneous display, wherein the second computing device is different from the first computing device.

15. The system of claim 14 , wherein the first computing device is associated with a customer and the second computing device is associated with a merchant.

16. The system of claim 11 , wherein the selection of the item for display and the positioning signal are re

Assignees

• Shopify Inc

Inventors

• Beauchamp Daniel

Classifications

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06Q30/06

  Buying, selling or leasing transactions · CPC title

• G06Q30/0601

  Electronic shopping [e-shopping] · CPC title

• G06Q30/0641

  utilising user interfaces specially adapted for shopping · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title