Object relationship estimation from a 3D semantic mesh

Patent metadata

Field	Value
Publication number	US-12175162-B2
Application number	US-202016984406-A
Country	US
Kind code	B2
Filing date	Aug 4, 2020
Priority date	Sep 10, 2019
Publication date	Dec 24, 2024
Grant date	Dec 24, 2024

Abstract

Implementations disclosed herein provide systems and methods that determine relationships between objects based on an original semantic mesh of vertices and faces that represent the 3D geometry of a physical environment. Such an original semantic mesh may be generated and used to provide input to a machine learning model that estimates relationships between the objects in the physical environment. For example, the machine learning model may output a graph of nodes and edges indicating that a vase is on top of a table or that a particular instance of a vase, V1, is on top of a particular instance of a table, T1.

First claim

What is claimed is:

1. A method comprising:
at an electronic device having a processor:
generating a semantic mesh of vertices defining faces representing 3D positions of surfaces of objects of a physical environment, at least some of the vertices having semantic labels identifying object type;
transforming the semantic mesh into a graph representing the semantic mesh, wherein vertices of the semantic mesh are represented by nodes of the graph and positional relationships between the nodes are determined from the 3D positions and represented by edges of the graph;
determining a reduced graph representation by combining nodes of the graph connected by edges and having a same semantic label;
identifying relative spatial relationships between the objects using the reduced graph representation; and
providing a computer-generated reality (CGR) environment that includes the objects, wherein the CGR environment is provided based on the relative spatial relationships between the objects.

2. The method of claim 1 further comprising determining the reduced graph representation by removing nodes in the graph.

3. The method of claim 2 , wherein nodes are removed based on removing edges between nodes having a same semantic label.

4. The method of claim 2 , wherein nodes are removed by:
determining that a first node and a second node are connected by an edge;
determining that the first node and second node have a same semantic label;
generating a combined node by combining the first node and the second node; and
merging duplicate edges as a result of combining nodes.

5. The method of claim 4 , wherein the combined node identifies an average position of a position associated with the first node and a position associated with the second node.

6. The method of claim 4 , wherein the combined node identifies a first position of the first node and a second position of the second node.

7. The method of claim 1 , wherein at least some of the nodes are semantically labelled floor, table, chair, wall, or ceiling.

8. The method of claim 1 , wherein the graph comprises edges connecting nodes associated with a same semantic label and edges connecting nodes associated with different semantic labels.

9. The method of claim 1 , wherein identifying the relative spatial relationships comprises identifying probabilities of the objects being associated by the relative spatial relationships.

10. The method of claim 1 , wherein a relationship of the relative spatial relationships identifies:
a first object on top of a second object;
the first object next to the second object;
the first object facing the second object; or
the first object attached to the second object.

11. The method of claim 1 , wherein identifying the relative spatial relationships between the objects comprises inputting into a machine learning model:
the reduced graph representation;
an image of the physical environment; and
a pose associated with a viewpoint in the physical environment.

12. The method of claim 1 further comprising providing a graph representing the objects and the relationships.

13. The method of claim 1 further comprising:
receiving input to position a virtual object in the environment that includes the objects; and
determining a position for the virtual object in the CGR environment based on the input and the relationships between the objects.

14. The method of claim 1 further comprising updating object classification labels of nodes of the reduced graph representation of the graph using a machine learning model.

15. The method of claim 1 , wherein identifying the relative spatial relationships between the objects comprises using a machine learning model that is trained using training data, the training data generated by:
modeling a plurality of meshes for separate objects of a synthetic environment, the separate objects associated with object types and the meshes associated with semantic labels;
determining a volume representation based on the plurality of meshes;
determining a combined mesh based on the volume representation; and
determining relationships between the separate objects.

16. A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
generating a semantic mesh of vertices defining faces representing 3D positions of surfaces of objects of a physical environment, at least some of the vertices having semantic labels identifying object type;
transforming the semantic mesh into a graph representing the semantic mesh, wherein vertices of the semantic mesh are represented by nodes of the graph and positional relationships between the nodes are determined from the 3D positions and represented by edges of the graph;
determining a reduced graph representation by combining nodes of the graph connected by edges and having a same semantic label;
identifying relative spatial relationships between the objects using the reduced graph representation; and
providing a computer-generated reality (CGR) environment that includes the objects, wherein the CGR environment is provided based on the relative spatial relationships between the objects.

17. The system of claim 16 , wherein the operations further comprise determining the reduced graph representation by removing nodes in the graph.

18. The system of claim 17 , wherein nodes are removed based on removing edges between nodes having a same semantic label.

19. The system of claim 17 , wherein nodes are removed by:
determining that a first node and a second node are connected by an edge;
determining that the first node and second node have a same semantic label;
generating a combined node by combining the first node and the second node; and
merging duplicate edges as a result of combining nodes.

20. A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
generating a semantic mesh of vertices defining faces representing 3D positions of surfaces of objects of a physical environment, at least some of the vertices having semantic labels identifying object type;
transforming the semantic mesh into a graph representing the semantic mesh, wherein vertices of the semantic mesh are represented by nodes of the graph and positional relationships between the nodes are determined from the 3D positions and represented by edges of the graph; and
determining a reduced graph representation by combining nodes of the graph connected by edges and having a same semantic label;
identifying relative spatial relationships between the objects using the reduced graph representation; and
providing a computer-generated reality (CGR) environment that includes the objects, wherein the CGR environment is provided based on the relative spatial relationships between the objects.

Assignees

• Apple Inc

Inventors

• Blechschmidt Angela
• Ulbricht Daniel
• Elafifi Omar

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06V10/768

  using context analysis, e.g. recognition aided by known co-occurring patterns · CPC title