Beyond shared hierarchies: deep multitask learning through soft layer ordering

Patent metadata

Field	Value
Publication number	US-11250314-B2
Application number	US-201816172660-A
Country	US
Kind code	B2
Filing date	Oct 26, 2018
Priority date	Oct 27, 2017
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

The technology disclosed identifies parallel ordering of shared layers as a common assumption underlying existing deep multitask learning (MTL) approaches. This assumption restricts the kinds of shared structure that can be learned between tasks. The technology disclosed demonstrates how direct approaches to removing this assumption can ease the integration of information across plentiful and diverse tasks. The technology disclosed introduces soft ordering as a method for learning how to apply layers in different ways at different depths for different tasks, while simultaneously learning the layers themselves. Soft ordering outperforms parallel ordering methods as well as single-task learning across a suite of domains. Results show that deep MTL can be improved while generating a compact set of multipurpose functional primitives, thus aligning more closely with our understanding of complex real-world processes.

First claim

What is claimed is:

1. A neural network-based system coupled to memory and running on one or more parallel processors, comprising:
an encoder generator which generates an encoder by accessing a set of processing submodules defined for the neural network-based system, constructing clones of the set of processing submodules, and arranging the clones in the encoder in a clone sequence starting from a lowest depth and continuing to a highest depth, wherein the clones in the encoder are shared by a plurality of classification tasks;
a feeder which feeds input data for a particular one of each of the plurality of classification tasks to each one of processing submodules in a first clone at the lowest depth in the clone sequence to produce an output encoding from each one of the processing submodules in the first clone;
a scaler generator which generates a scaler for the first clone, wherein the scaler applies task-specific, depth-specific, and processing submodule-specific scaling values to respective output encodings of the processing submodules in the first clone to produce a scaled output encoding for each one of the processing submodules in the first clone;
an accumulator that mixes respective scaled output encodings of the processing submodules in the first clone to produce an accumulated output encoding for the first clone;
a forwarder that transmits the accumulated output encoding for the first clone as input to each one of processing submodules in a successive clone at a successive depth in the clone sequence;
a controller that iteratively invokes the scaler generator, the accumulator, and the forwarder to, respectively, produce a scaled output encoding for each one of the processing submodules in the successive clone, produce an accumulated output encoding for the successive clone, and transmit the accumulated output encoding for the successive clone as input to each one of processing submodules in another successive clone at another successive depth in the clone sequence until an accumulated output encoding is produced for a final clone at the highest depth in the clone sequence;
a decoder selector which selects, from among numerous decoders, a decoder that is specific to each of a particular one of the classification tasks and transmits the accumulated output encoding produced for the final clone as input to the selected decoder; and
the selected decoder which processes the accumulated output encoding and produces classification scores for classes defined for each of the particular one of the classification tasks.

2. The neural network-based system of claim 1 , wherein the scaler is a three-dimensional tensor that is learned using a gradient-update technique based on backpropagation.

3. The neural network-based system of claim 1 , wherein the scaling values are scalar values that augment or diminish respective magnitudes of the output encodings.

4. The neural network-based system of claim 3 , wherein the scalar values are softmax values that sum to unity.

5. The neural network-based system of claim 3 , wherein the scalar values are sigmoid values between zero and unity.

6. The neural network-based system of claim 3 , wherein the scalar values are continuous values normalized between a lowest value and a highest value.

7. The neural network-based system of claim 1 , wherein the processing submodules in the set have at least one different global topology hyperparameter, global operational hyperparameter, local topology hyperparameter, and/or local operational hyperparameter.

8. The neural network-based system of claim 1 , wherein the encoder is a convolutional neural network and the processing submodules are convolution layers interspersed with activation and/or normalization functions.

9. The neural network-based system of claim 1 , wherein the encoder is a recurrent neural network and the processing submodules are recurrent layers interspersed with activation and/or normalization functions.

10. The neural network-based system of claim 1 , wherein each decoder further comprises at least one decoder layer and at least one classification layer.

11. The neural network-based system of claim 10 , wherein the decoder is a fully-connected neural network and the decoder layer is a fully-connected layer.

12. The neural network-based system of claim 10 , wherein the classification layer is a sigmoid classifier.

13. The neural network-based system of claim 10 , wherein the classification layer is a softmax classifier.

14. A neural network-implemented method of soft ordering, including:
generating an encoder by accessing a set of processing submodules defined for a neural network-based model, constructing clones of the set of processing submodules, and arranging the clones in the encoder in a clone sequence starting from a lowest depth and continuing to a highest depth, wherein the clones in the encoder are shared by a plurality of classification tasks;
feeding input data for a particular one of each of the plurality of classification tasks to each one of processing submodules in a first clone at the lowest depth in the clone sequence to produce an output encoding from each one of the processing submodules in the first clone;
generating a scaler for the first clone, wherein the scaler applies task-specific, depth-specific, and processing submodule-specific scaling values to respective output encodings of the processing submodules in the first clone to produce a scaled output encoding for each one of the processing submodules in the first clone;
mixing respective scaled output encodings of the processing submodules in the first clone to produce an accumulated output encoding for the first clone;
transmitting the accumulated output encoding for the first clone as input to each one of processing submodules in a successive clone at a successive depth in the clone sequence;
iterating the scaler generation, the mixing, and the transmitting to, respectively, produce a scaled output encoding for each one of the processing submodules in the successive clone, produce an accumulated output encoding for the successive clone, and transmit the accumulated output encoding for the successive clone as input to each one of processing submodules in another successive clone at another successive depth in the clone sequence until an accumulated output encoding is produced for a final clone at the highest depth in the clone sequence;
selecting, from among numerous decoders, a decoder that is specific to each of a particular one of the classification tasks and transmitting the accumulated output encoding produced for the final clone as input to the selected decoder; and
processing the accumulated output encoding through the selected decoder to produce classification scores for classes defined for each of the particular one of the classification tasks.

15. The neural network-implemented method of claim 14 , wherein the scaler is a three dimensional tensor that is learned using a gradient-update technique based on backpropagation.

16. The neural network-implemented method of claim 14 , wherein the scaling values are scalar values that augment or diminish respective magnitudes of the output encodings.

17. The neural network-implemented method of claim 16 , wherein the scalar values are softmax values that sum to unity.

18. The neural network-implemented method of claim 16 , wherein the scalar values are sigmoid values between zero and unity.

19. The neural network-implemented method of claim 16 , wherein the scalar values are continuous values normalized between a

Assignees

• Cognizant Tech Solutions U S Corporation

Inventors

• Meyerson Elliot
• Miikkulainen Risto

Classifications

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N3/048 Primary

  Activation functions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title