Segmenting three-dimensional shapes into labeled component shapes

Patent metadata

Field	Value
Publication number	US-10467760-B2
Application number	US-201715440572-A
Country	US
Kind code	B2
Filing date	Feb 23, 2017
Priority date	Feb 23, 2017
Publication date	Nov 5, 2019
Grant date	Nov 5, 2019

Abstract

This disclosure involves generating and outputting a segmentation model using 3D models having user-provided labels and scene graphs. For example, a system uses a neural network learned from the user-provided labels to transform feature vectors, which represent component shapes of the 3D models, into transformed feature vectors identifying points in a feature space. The system identifies component-shape groups from clusters of the points in the feature space. The system determines, from the scene graphs, parent-child relationships for the component-shape groups. The system generates a segmentation hierarchy with nodes corresponding to the component-shape groups and links corresponding to the parent-child relationships. The system trains a point classifier to assign feature points, which are sampled from an input 3D shape, to nodes of the segmentation hierarchy, and thereby segment the input 3D shape into component shapes.

First claim

The invention claimed is:

1. A method comprising:
accessing, from a memory device, training data comprising a set of 3D models with a set of user-provided labels and a set of user-provided scene graphs, wherein a 3D model in the set of 3D models comprises a set of component shapes tagged with a subset of the set of user-provided labels, wherein the component shapes of the 3D model have model parent-child relationships described by a subset of the user-provided scene graphs;
generating, by a processing device and from the training data, a segmentation model, wherein generating the segmentation model includes:
transforming, with a neural network model learned from the user-provided labels, feature vectors representing the component shapes of the 3D models into transformed feature vectors identifying points in a multi-dimensional feature space;
grouping the points in the multi-dimensional feature space into clusters;
detecting, based on the user-provided scene graphs, an inconsistency in a first cluster from the clusters, wherein the inconsistency occurs due to the first cluster comprising a first point corresponding to a first component shape and a second point corresponding to a second component shape, and wherein the first component shape is identified as a child of the second component shape in at least one user-provided scene graph of the set of user-provided scene graphs;
modifying, based on the detected inconsistency, the first cluster such that the first point is included in a second cluster;
deriving component-shape groups from the modified cluster, wherein each component-shape group includes a respective subset of the component shapes from the 3D models and is associated with a respective label obtained from a respective corresponding subset of the user-provided labels; and
training a segmentation algorithm of the segmentation model based on the component-shape groups identified from the clusters; and

outputting, by the processing device, the segmentation model having the trained segmentation algorithm.

2. The method of claim 1 , further comprising training the neural network model by performing, by the processing device, operations comprising:
selecting, from the feature vectors representing the component shapes, a first training feature vector and a second training feature vector, wherein the first training feature vector represents a component shape having a training label from the user-provided labels and the second training feature vector represents an additional component shape having the training label;
transforming, with the neural network model, the first training feature vector and the second training feature vector into a first transformed feature vector and a second transformed feature vector, respectively;
determining, based on the training label and a loss function, that a first distance between the first transformed feature vector and the second transformed feature vector exceeds a threshold;
adjusting the neural network model based on the first transformed feature vector and the second transformed feature vector exceeds the threshold;
transforming, with the neural network model as adjusted, the first training feature vector and the second training feature vector into a first adjusted transformed feature vector and a second adjusted transformed feature vector, respectively; and
determining, based on the training label and the loss function, that a second distance between the first adjusted transformed feature vector and the second adjusted transformed feature vector is less than the threshold.

3. The method of claim 1 , wherein the training data comprises an un-labeled component shape lacking training labels in the user-provided labels, wherein the method further comprises:
transforming, with the neural network model trained from the user-provided labels, an additional feature vector representing the un-labeled component shape into an additional transformed feature vector in the multi-dimensional feature space;
identifying, for the additional transformed feature vector, a neighboring transformed feature vector that is associated with a labeled component shape; and
applying a label from the labeled component shape to the un-labeled component shape.

4. The method of claim 1 , wherein modifying the first cluster comprises:
selecting the second cluster based on the second cluster having a threshold number of additional points within a threshold distance of the first point, wherein the threshold number of additional points being within the threshold distance indicates the first component shape has a threshold similarity to additional component shapes corresponding to the additional points; and
assigning the first point to the second cluster.

5. The method of claim 1 , wherein modifying the first cluster comprises:
selecting the second cluster based on the second cluster having a threshold number of points corresponding to child component shapes that are identified, in the user-provided scene graphs, as children of parent component shapes corresponding to points in the first cluster; and
assigning the first point to the second cluster.

6. The method of claim 1 , wherein training the segmentation algorithm comprises:
identifying a component shape that is a parent of an additional component shape;
identifying, from the user-provided labels, a first label of the component shape that is the parent and a second label of the additional component shape; and
iteratively modifying a point classifier such that an energy function associated with the point classifier is minimized, wherein the energy function indicates a probability of the point classifier applying the second label rather than the first label to the component shape that is the parent.

7. The method of claim 1 , wherein training the segmentation algorithm comprises:
determining, from the user-provided scene graphs, group parent-child relationships for the component-shape groups, wherein a group parent-child relationship between a first component-shape group and a second component-shape group is determined based on a number of first component shapes in the first component-shape group being identified by the user-provided scene graphs as parents of second component shapes in the second component-shape group;
generating a segmentation hierarchy comprising nodes that respectively correspond to the component-shape groups and links among the nodes that respectively correspond to the group parent-child relationships; and
training, based on the clusters and feature points sampled from the component shapes, a point classifier to recognize a correspondence between an input feature point sampled from an input 3D shape and a node of the segmentation hierarchy, wherein the segmentation model includes the trained point classifier and the segmentation hierarchy.

8. The method of claim 7 , further comprising segmenting an input 3D shape by performing operations comprising:
generating, using the trained point classifier and based on the segmentation hierarchy from the segmentation model, an initial segmentation hierarchy and an initial set of label for component shapes of the input 3D shape; and
optimizing the initial segmentation hierarchy by maximizing an energy function having a first term indication a probability of a particular component shape having a first label, a second term indicating constraints applied to component shapes in a common level of the initial segmentation hierarchy, and a third term indicating constraints applied to a component shape across different levels of the initial segmentation hierarchy,
wherein maximizing the energy function (i) generates an output segmentation hierarchy from the initial segmentation hierarchy by removing one or more pare

Assignees

• Adobe Inc

Inventors

• Kim Vladimir
• Hertzmann Aaron
• Yumer Mehmet
• Yi Li

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06T7/162 Primary

  involving graph-based methods · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/11

  Region-based segmentation · CPC title