Systems and methods for generating augmented reality content based on distorted three-dimensional models

Patent metadata

Field	Value
Publication number	US-11836877-B2
Application number	US-202217845387-A
Country	US
Kind code	B2
Filing date	Jun 21, 2022
Priority date	Sep 23, 2020
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

Systems and methods for generating AR content to virtually alter a real-world item are provided. The real-world item may be a deformable item including one or more flexible surfaces. According to an embodiment, a method includes determining the relative positions of a plurality of points on the real-world item. Based on the relative positions of the plurality of points, a 3D model of a virtual item is distorted to produce a distorted 3D model of the virtual item. The distorted 3D model may have substantially the same shape as the real-world item. AR content is then generated based on the distorted three-dimensional model.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
determining relative positions of a plurality of points on a real-world item, the relative positions of the plurality of points on the real-world item corresponding to a shape of the real-world item;
modifying a three-dimensional model of a virtual item, the modifying including repositioning points on the three-dimensional model to be in relative positions corresponding to the relative positions of the plurality of points on the real-world item; and
generating augmented reality (AR) content for presentation, the AR content being based on the modified three-dimensional model.

2. The computer-implemented method of claim 1 , wherein the three-dimensional model of the virtual item yielded by the repositioning the points on the three-dimensional model has a shape corresponding to the shape of the real-world item.

3. The computer-implemented method of claim 1 , wherein the virtual item provides a visual appearance for the real-world item in the AR content.

4. The computer-implemented method of claim 1 , wherein the virtual item corresponds to the real-world item.

5. The computer-implemented method of claim 1 , wherein generating the AR content includes rendering the modified three-dimensional model, and wherein:
the modified three-dimensional model is rendered from a viewpoint corresponding to a captured image of the real-world item; and
generating the AR content further includes compositing the rendering with the captured image to overlay the real-world item.

6. The computer-implemented method of claim 1 , wherein generating the AR content includes rendering the modified three-dimensional model, the method further comprising:
determining a position of a user relative to the real-world item, wherein the modified three-dimensional model is rendered based on the position of the user.

7. The computer-implemented method of claim 1 , wherein the real-world item includes a flexible surface.

8. The computer-implemented method of claim 1 , wherein the real-world item is or includes a garment.

9. The computer-implemented method of claim 1 , wherein:
the real-world item comprises a plurality of optical markers, each of the plurality of optical markers being at a respective different one of the plurality of points on the real-world item; and
determining the relative positions of the plurality of points on the real-world item comprises determining relative positions of the plurality of optical markers by performing image analysis on at least one image of the real-world item to detect the optical markers.

10. The computer-implemented method of claim 1 , wherein:
the relative positions of the plurality of points are first relative positions of the plurality of points, the modified three-dimensional model of the virtual item is a first modified three-dimensional model of the virtual item, the shape is a first shape, and the AR content is first AR content; and
the method further comprises:
determining second relative positions of the plurality of points on the real-world item, the second relative positions corresponding to the real-world item in a second shape;
modifying the three-dimensional model of a virtual item to produce a second modified three-dimensional model, the modifying including repositioning points on the three-dimensional model to be in relative positions corresponding to the second relative positions of the plurality of points on the real-world item; and
generating second AR content for presentation, the second AR content being based on the second modified three-dimensional model.

11. The computer-implemented method of claim 1 , wherein:
the relative positions of the plurality of points on the real-world item are based on measurements obtained from a device; and
the AR content is for presentation at that device.

12. The computer-implemented method of claim 1 , wherein the three-dimensional model of the virtual item is a first three-dimensional model of the virtual item, the relative positions of the plurality of points are first relative positions of the plurality of points, and the AR content is first AR content, the method further comprising:
obtaining an instruction to replace the first three-dimensional model of the virtual item with a second three-dimensional model of the virtual item, the instruction originating from user input at a device;
determining second relative positions of the plurality of points on the real-world item, the second relative positions of the plurality of points on the real-world item corresponding to a second shape of the real-world item;
modifying the second three-dimensional model of a virtual item, the modifying including repositioning points on the second three-dimensional model to be in relative positions corresponding to the second relative positions of the plurality of points on the real-world item;
generating second AR content for presentation, the second AR content being based on the modified second three-dimensional model.

13. A system comprising:
memory to store relative positions of a plurality of points on a real-world item, the relative positions of the plurality of points on the real-world item corresponding to a shape of the real-world item; and
at least one processor to:
modify a three-dimensional model of a virtual item, the modifying including repositioning points on the three-dimensional model to be in relative positions corresponding to the relative positions of the plurality of points on the real-world item; and
generate augmented reality (AR) content for presentation, the AR content being based on the modified three-dimensional model.

14. The system of claim 13 , wherein the three-dimensional model of the virtual item yielded by the repositioning the points on the three-dimensional model has a shape corresponding to the shape of the real-world item.

15. The system of claim 13 , wherein the virtual item provides a visual appearance for the real-world item in the AR content.

16. The system of claim 13 , wherein the virtual item corresponds to the real-world item.

17. The system of claim 13 , wherein:
the at least one processor is further to render the modified three-dimensional model;
the modified three-dimensional model is rendered from a viewpoint corresponding to a captured image of the real-world item; and
the AR content includes the render composited with the captured image to overlay the real-world item.

18. The system of claim 13 , wherein:
the relative positions of the plurality of points are first relative positions of the plurality of points, the modified model of the virtual item is a first modified model of the virtual item, and the AR content is first AR content;
the first relative positions of the plurality of points correspond to the real-world item in a first shape;
the memory is further to store second relative positions of the plurality of points on the real-world item, the second relative positions corresponding to the real-world item in a second shape; and
the at least one processor is further to:
modify the three-dimensional model of a virtual item to produce a second modified three-dimensional model, the modifying including repositioning points on the three-dimensional model to be in relative positions corresponding to the second relative positions of the plurality of points on the real-world item; and
generate second AR content for presentation, the second AR content being based on the second modified three-dimensional model.

19. The system of claim 13 ,

Assignees

• Shopify Inc

Inventors

• Haapoja Juho Mikko
• Delgado Byron Leonel
• Leroux Stephan
• Beauchamp Daniel
• Lalani Maas Mansoor Ali

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06N20/00

  Machine learning · CPC title

• G06Q30/0643

  graphically representing goods, e.g. 3D product representation · CPC title

• G06T15/20

  Perspective computation · CPC title

• G06T2207/20081

  Training; Learning · CPC title