Object Relationship Estimation From A 3D Semantic Mesh

Patent metadata

Field	Value
Publication number	US-2021073429-A1
Application number	US-202016984406-A
Country	US
Kind code	A1
Filing date	Aug 4, 2020
Priority date	Sep 10, 2019
Publication date	Mar 11, 2021
Grant date	—

Abstract

Implementations disclosed herein provide systems and methods that determine relationships between objects based on an original semantic mesh of vertices and faces that represent the 3D geometry of a physical environment. Such an original semantic mesh may be generated and used to provide input to a machine learning model that estimates relationships between the objects in the physical environment. For example, the machine learning model may output a graph of nodes and edges indicating that a vase is on top of a table or that a particular instance of a vase, V1, is on top of a particular instance of a table, T1.

First claim

What is claimed is:

1 . A method comprising:
at an electronic device having a processor: generating a semantic mesh of vertices and faces, the faces representing surfaces of objects of a physical environment and at least some of the vertices having semantic labels identifying object type; transforming the semantic mesh into a graph representing the semantic mesh, wherein vertices of the semantic mesh are represented by nodes of the graph; and identifying relationships between the objects using a machine learning model that inputs a representation of the graph of the semantic mesh.

2 . The method of claim 1 further comprising determining the representation of the graph of the semantic mesh by removing nodes in the graph.

3 . The method of claim 2 , wherein nodes are removed based on removing edges between nodes having a same semantic label.

4 . The method of claim 2 , wherein nodes are removed by:
determining that a first node and a second node are connected by an edge; determining that the first node and second node have a same semantic label; generating a combined node by combining the first node and the second node; and merging duplicate edges as a result of combining nodes.

5 . The method of claim 4 , wherein the combined node identifies an average position of the first node and the second node.

6 . The method of claim 4 , wherein the combined node identifies a first position of the first node and a second position of the second node.

7 . The method of claim 1 , wherein at least some of the nodes are semantically labelled floor, table, chair, wall, or ceiling.

8 . The method of claim 1 , wherein the graph comprises edges connecting nodes associated with a same semantic label and edges connecting nodes associated with different semantic labels.

9 . The method of claim 1 , wherein identifying relationships comprises identifying probabilities of the objects being associated by the relationships.

10 . The method of claim 1 , wherein a relationship of the relationships identifies:
a first object on top of a second object; the first object next to the second object; the first object facing the second object; or the first object attached to the second object.

11 . The method of claim 1 , wherein the machine learning model also uses as input:
an image of the physical environment; or a pose associated with a viewpoint in the physical environment

12 . The method of claim 1 further comprising providing a graph representing the objects and the relationships.

13 . The method of claim 1 further comprising:
receiving input to position a virtual object in a computer-generated reality (CGR) environment that includes the objects;
determining a position for the virtual object in the CGR environment based on the input and the relationships between the objects; and
providing the CGR environment.

14 . The method of claim 1 further comprising updating object classification labels of nodes of the representation of the graph using a machine learning model.

15 . The method of claim 1 , wherein the machine learning model is trained using training data, the training data generated by:
modeling a plurality of meshes for separate objects of a synthetic environment, the separate objects associated with object types and the meshes associated with semantic labels; determining a volume representation based on the plurality of meshes; determining a combined mesh based on the volume representation; determining relationships between the separate objects.

16 . A system comprising:
a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising: generating a semantic mesh of vertices and faces, the faces representing surfaces of objects of a physical environment and at least some of the vertices having semantic labels identifying object type; transforming the semantic mesh into a graph representing the semantic mesh, wherein vertices of the semantic mesh are represented by nodes of the graph; and identifying relationships between the objects using a machine learning model that inputs a representation of the graph of the semantic mesh.

17 . The system of claim 16 , wherein the operations further comprise determining the representation of the graph of the semantic mesh by removing nodes in the graph.

18 . The system of claim 17 , wherein nodes are removed based on removing edges between nodes having a same semantic label.

19 . The system of claim 17 , wherein nodes are removed by:
determining that a first node and a second node are connected by an edge; determining that the first node and second node have a same semantic label; generating a combined node by combining the first node and the second node; and merging duplicate edges as a result of combining nodes.

20 . A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
generating a semantic mesh of vertices and faces, the faces representing surfaces of objects of a physical environment and at least some of the vertices having semantic labels identifying object type; transforming the semantic mesh into a graph representing the semantic mesh, wherein vertices of the semantic mesh are represented by nodes of the graph; and identifying relationships between the objects using a machine learning model that inputs a representation of the graph of the semantic mesh.

Assignees

• Apple Inc

Inventors

• Blechschmidt Angela
• Ulbricht Daniel
• Elafifi Omar

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title