System and method for implementing modular universal reparameterization for deep multi-task learning across diverse domains

Patent metadata

Field	Value
Publication number	US-11669716-B2
Application number	US-202016817153-A
Country	US
Kind code	B2
Filing date	Mar 12, 2020
Priority date	Mar 13, 2019
Publication date	Jun 6, 2023
Grant date	Jun 6, 2023

Abstract

A process for training and sharing generic functional modules across multiple diverse (architecture, task) pairs for solving multiple diverse problems is described. The process is based on decomposing the general multi-task learning problem into several fine-grained and equally-sized subproblems, or pseudo-tasks. Training a set of (architecture, task) pairs then corresponds to solving a set of related pseudo-tasks, whose relationships can be exploited by shared functional modules. An efficient search algorithm is introduced for optimizing the mapping between pseudo-tasks and the modules that solve them, while simultaneously training the modules themselves.

First claim

The invention claimed is:

1. A machine-learning process for training and sharing generic functional modules across multiple diverse (architecture, task) pairs for solving multiple diverse problems, comprising:
decomposing, by one or more specially programmed processors, each of the multiple (architecture, task) pairs into equally sized pseudo-tasks;
aligning, by the one or more specially programmed processors pseudo-tasks across the multiple diverse architectures, wherein aligning pseudo-tasks comprises optimizing a mapping between the pseudo-tasks and the functional modules of the diverse architectures that solve the pseudo-tasks; and
sharing, by the one or more specially programmed processors, learned parameters across the aligned pseudo-tasks, wherein each diverse architecture is preserved in performance of its paired task.

2. The process according to claim 1 , wherein the diverse architectures include core layers selected from the following group consisting of 2D convolutional, LSTM, 1D convolutional and Dense.

3. The process according to claim 1 , wherein the optimizing uses a stochastic algorithm.

4. The process according to claim 1 , wherein the sharing by the one or more specially programmed processors learned parameters across the aligned pseudo-tasks is implemented using factorization.

5. The process according to claim 1 , wherein the multiple diverse problems are selected from the group consisting of a vision problem, a sorting problem, a natural language processing problem, a speech problem, a biological problem, a geological problem and an astronomical problem.

6. A machine-learning process for training and sharing functional modules across diverse architectures for performing diverse tasks without changing functional forms of underlying predictive models, comprising:
decomposing, by one or more specially programmed processors, each parameter set for each predictive model into parameter blocks, wherein a parameter block is parameterized by a module;
aligning, by the one or more specially programmed processors, pseudo-tasks in accordance with associated parameter sets across the multiple diverse architectures, wherein the aligning includes optimizing a mapping between the pseudo-tasks and the functional modules that solve the pseudo-tasks; and
sharing, by the one or more specially programmed processors, modules across the diverse architectures in accordance with a predetermined alignment, wherein the diverse architectures perform diverse tasks and the sharing of modules improves performance in each diverse task.

7. The process according to claim 6 , wherein the diverse architectures include core layers selected from the following group consisting of 2D convolutional, LSTM, 1D convolutional and Dense.

8. The process according to claim 6 , wherein the optimizing uses a stochastic algorithm.

9. The process according to claim 6 , wherein the sharing by the one or more specially programmed processors includes sharing learned parameters across the aligned pseudo-tasks.

10. The process according to claim 9 , wherein sharing learned parameters across the aligned pseudo-tasks is implemented using factorization.

11. The process according to claim 6 , wherein the multiple diverse tasks are selected from the group consisting of a vision-related task, a sorting-related task, a natural language processing-related task, a speech-related task, a biological-related task, a geological-related task and an astronomical-related task.

12. A computer-implemented machine learning process for training and sharing generic functional modules across multiple diverse (architecture, task) pairs for solving multiple diverse problems, comprising:
means for decomposing, by one or more specially programmed processors, each of the multiple (architecture, task) pairs into equally sized pseudo-tasks;
means for aligning, by the one or more specially programmed processors, pseudo-tasks across the multiple diverse architectures, wherein aligning pseudo-tasks comprises optimizing a mapping between the pseudo-tasks and the functional modules of the diverse architectures that solve the pseudo-tasks; and
means for sharing, by the one or more specially programmed processors, learned parameters across the aligned pseudo-tasks, wherein each diverse architecture is preserved in performance of its paired task.

13. The process according to claim 12 , wherein the means of optimizing is a stochastic algorithm.

14. The process according to claim 12 , wherein the means for sharing by the one or more specially programmed processors learned parameters across the aligned pseudo-tasks includes factorization.

Assignees

• Cognizant Tech Solutions U S Corporation
• Cognizant Tech Solutions U S Corp

Inventors

• Meyerson Elliot
• Miikkulainen Risto

Classifications

• G06N3/086

  using evolutionary algorithms, e.g. genetic algorithms or genetic programming · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/047 Primary

  Probabilistic or stochastic networks · CPC title

• G06N3/09

  Supervised learning · CPC title