Semi-supervised learning using clustering as an additional constraint

Patent metadata

Field	Value
Publication number	US-11954881-B2
Application number	US-201916513991-A
Country	US
Kind code	B2
Filing date	Jul 17, 2019
Priority date	Aug 28, 2018
Publication date	Apr 9, 2024
Grant date	Apr 9, 2024

Abstract

In some implementations a neural network is trained to perform a main task using a clustering constraint, for example, using both a main task training loss and a clustering training loss. Training inputs are inputted into a main task neural network to produce output labels predicting locations of the parts of the objects in the training inputs. Data from pooled layers of the main task neural network is inputted into a clustering neural network. The main task neural network and the clustering neural network are trained based on a main task loss from the main task neural network and a clustering loss from the clustering neural network. The main task loss is determined by comparing differences between the output labels and the training labels. The clustering loss encourages the clustering network to learn to label the parts of the objects individually, e.g., to learn groups corresponding to the object parts.

First claim

What is claimed is:

1. A method, comprising:
at an electronic device with one or more processors:
obtaining a training set of training inputs and corresponding training labels, the training labels identifying known locations of parts of objects in the training inputs;
inputting the training inputs into a main task neural network to produce output labels predicting locations of the parts of the objects in the training inputs;
inputting data from pooled layers of the main task neural network into a clustering neural network to identify clusters comprising a first group of features in a first subset of the pooled layers that correspond with a first part of at least one object of the objects and a second group of features in a second subset of the pooled layers that correspond with a second part of the at least one object of the objects, wherein the first subset is different than the second subset of pooled; and
training the main task neural network and the clustering neural network based on a main task loss from the main task neural network and a clustering loss from the clustering neural network.

2. The method of claim 1 further comprising:
inputting additional inputs into the main task neural network to produce additional output labels and corresponding confidence values;
selecting, based on the confidence values, an automatically-labeled training set of data comprising a subset of the additional inputs and a corresponding subset of the additional output labels; and
further training the main task neural network and the clustering neural network using the automatically-labeled training set of data.

3. The method of claim 1 further comprising determining the main task loss by comparing the output labels and the training labels.

4. The method of claim 1 further comprising determining the main task loss using learned quality assurance metrics.

5. The method of claim 1 wherein the clustering loss is configured to cause the clustering neural network to learn to label the parts of the objects individually.

6. The method of claim 1 , wherein the clustering loss is configured to cause the clustering neural network to learn groups corresponding to the parts of the objects.

7. The method of claim 1 , wherein the clustering neural network is trained to identify a first group of the pooled layers corresponding to a first pattern and a second group of the pooled layers corresponding to a second pattern.

8. The method of claim 1 , wherein the main task neural network and the clustering neural network are trained together using the main task loss and the clustering loss to cause clusters learnt by the clustering neural network to correspond to the parts of the objects.

9. The method of claim 1 , wherein the main task neural network and the clustering neural network are trained together using the main task loss and the clustering loss to cause similarity between sub-parts of feature maps across multiple images.

10. The method of claim 1 , wherein groups learned by the clustering neural network correspond to human body parts.

11. The method of claim 1 , wherein a number of groups learned by the clustering neural network corresponds to a number of the parts of the objects.

12. The method of claim 1 , wherein the main task neural network is trained for human pose estimation, wherein the parts of the objects correspond to parts of a skeleton representing human pose.

13. The method of claim 1 , wherein the main task neural network is trained for hand tracking, body tracking, or gaze tracking.

14. The method of claim 1 , wherein the main task neural network is trained for semantic segmentation of audio.

15. The method of claim 1 , wherein the main task neural network is trained for semantic segmentation of text.

16. The method of claim 1 further comprising integrating the main task neural network into an application stored on a non-transitory computer-readable medium.

17. A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
obtaining a training set of training inputs and corresponding training labels, the training labels identifying known locations of parts of objects in the training inputs;
inputting the training inputs into a main task neural network to produce output labels predicting locations of the parts of the objects in the training inputs;
inputting data from pooled layers of the main task neural network into a clustering neural network to identify clusters comprising a first group of features in a first subset of the pooled layers that correspond with a first part of at least one object of the objects and a second group of features in a second subset of the pooled layers that correspond with a second part of the at least one object of the objects, wherein the first subset is different than the second subset; and
training the main task neural network and the clustering neural network based on a main task loss from the main task neural network and a clustering loss from the clustering neural network.

18. The system of claim 17 , wherein the operations further comprise:
inputting additional inputs into the main task neural network to produce additional output labels and corresponding confidence values;
selecting, based on the confidence values, an automatically-labeled training set of data comprising a subset of the additional inputs and a corresponding subset of the additional output labels; and
further training the main task neural network and the clustering neural network using the automatically-labeled training set of data.

19. The system of claim 17 , wherein the operations further comprise determining the main task loss by comparing the output labels and the training labels.

20. A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
obtaining a training set of training inputs and corresponding training labels, the training labels identifying known locations of parts of objects in the training inputs;
inputting the training inputs into a main task neural network to produce output labels predicting locations of the parts of the objects in the training inputs;
inputting data from pooled layers of the main task neural network into a clustering neural network to identify clusters comprising a first group of features in a first subset of the pooled layers that correspond with a first part of at least one object of the objects and a second group of features in a second subset of the pooled layers that correspond with a second part of the at least one object of the objects, wherein the first subset is different than the second subset; and
training the main task neural network and the clustering neural network based on a main task loss from the main task neural network and a clustering loss from the clustering neural network.

Assignees

• Apple Inc

Inventors

• Meier Peter
• Batra Tanmay

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06F18/2115

  by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination · CPC title