Semantic texture mapping system

Patent metadata

Field	Value
Publication number	US-12039658-B2
Application number	US-202217752331-A
Country	US
Kind code	B2
Filing date	May 24, 2022
Priority date	Apr 1, 2019
Publication date	Jul 16, 2024
Grant date	Jul 16, 2024

Abstract

A semantic texture map system to generate a semantic texture map based on a 3D model that comprises a plurality of vertices that include coordinate that indicate positions of the plurality of vertices, a UV map, and a semantic segmentation image that comprises a set of semantic labels.

First claim

What is claimed is:

1. A method comprising:
causing display of image data at a client device, the image data corresponding with a location, and including a display of an object at a position within the image data;
accessing a repository based on the location, the repository comprising a set of semantic labels that indicate an object class of the object;
retrieving Augmented-Reality (AR) content based on user profile data associated with a user of the client device and the display of the object; and
causing display of a presentation of the AR content at the position of the object within the image data at the client device based on the set of semantic labels that correspond with the object at the location.

2. The method of claim 1 , wherein the set of semantic labels include a semantic label that corresponds with a material parameter, and wherein the causing display of the presentation of the AR content is based on the material parameter.

3. The method of claim 2 , wherein the material parameter includes one or more of:
a roughness value;
a metallic value;
a specular value; and
a base color value.

4. The method of claim 1 , wherein the image data comprises metadata, and the method further comprises:
retrieving the AR content based on the user profile data associated with the user of the client device, and the metadata of the image data.

5. The method of claim 4 , wherein the metadata includes temporal data.

6. The method of claim 1 , wherein the image data comprises a set of image features, and the method further comprises:
identifying the location based on the set of image features.

7. The method of claim 6 , wherein the identifying the location based on the set of image features further comprises:
identifying the object based on the set of image features, the object corresponding with the location; and
identifying the location based on the object.

8. A system comprising:
a memory; and
at least one hardware processor coupled to the memory and comprising instructions that causes the system to perform operations comprising:
causing display of image data at a client device, the image data corresponding with a location, and including a display of an object at a position within the image data;
accessing a repository based on the location, the repository comprising a set of semantic labels that indicate an object class of the object;
retrieving Augmented-Reality (AR) content based on user profile data associated with a user of the client device and the display of the object; and
causing display of a presentation of the AR content at the position of the object within the image data at the client device based on the set of semantic labels that correspond with the object at the location.

9. The system of claim 8 , wherein the set of semantic labels include a semantic label that corresponds with a material parameter, and wherein the causing display of the presentation of the AR content is based on the material parameter.

10. The system of claim 9 , wherein the material parameter includes one or more of:
a roughness value;
a metallic value;
a specular value; and
a base color value.

11. The system of claim 8 , wherein the image data comprises metadata, and the method further comprises:
retrieving the AR content based on the user profile data associated with the user of the client device, and the metadata of the image data.

12. The system of claim 11 , wherein the metadata includes temporal data.

13. The system of claim 8 , wherein the image data comprises a set of image features, and the operations further comprise:
identifying the location based on the set of image features.

14. The system of claim 13 , wherein the identifying the location based on the set of image features further comprises:
identifying the object based on the set of image features, the object corresponding with the location; and
identifying the location based on the object.

15. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
causing display of image data at a client device, the image data corresponding with a location, and including a display of an object at a position within the image data;
accessing a repository based on the location, the repository comprising a set of semantic labels that indicate an object class of the object;
retrieving Augmented-Reality (AR) content based on user profile data associated with a user of the client device and the display of the object; and
causing display of a presentation of the AR content at the position of the object within the image data at the client device based on the set of semantic labels that correspond with the object at the location.

16. The non-transitory machine-readable storage medium of claim 15 , wherein the set of semantic labels include a semantic label that corresponds with a material parameter, and wherein the causing display of the presentation of the AR content is based on the material parameter.

17. The non-transitory machine-readable storage medium of claim 16 , wherein the material parameter includes one or more of:
a roughness value;
a metallic value;
a specular value; and
a base color value.

18. The non-transitory machine-readable storage medium of claim 15 , wherein the image data comprises metadata, and the method further comprises:
retrieving the AR content based on the user profile data associated with the user of the client device, and the metadata of the image data.

19. The non-transitory machine-readable storage medium of claim 18 , wherein the metadata includes temporal data.

20. The non-transitory machine-readable storage medium of claim 15 , wherein the image data comprises a set of image features, and the operations further comprise:
identifying the location based on the set of image features.

Assignees

• Snap Inc

Inventors

• Cowburn Piers
• Li David
• Müller Sandvik Isac Andreas
• Pan Qi

Classifications

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06V30/274

  Syntactic or semantic context, e.g. balancing · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title