System and Method For Implementing Modular Universal Reparameterization For Deep Multi-Task Learning Across Diverse Domains

Patent metadata

Field	Value
Publication number	US-2020293888-A1
Application number	US-202016817153-A
Country	US
Kind code	A1
Filing date	Mar 12, 2020
Priority date	Mar 13, 2019
Publication date	Sep 17, 2020
Grant date	—

Abstract

A process for training and sharing generic functional modules across multiple diverse (architecture, task) pairs for solving multiple diverse problems is described. The process is based on decomposing the general multi-task learning problem into several fine-grained and equally-sized subproblems, or pseudo-tasks. Training a set of (architecture,task) pairs then corresponds to solving a set of related pseudo-tasks, whose relationships can be exploited by shared functional modules. An efficient search algorithm is introduced for optimizing the mapping between pseudo-tasks and the modules that solve them, while simultaneously training the modules themselves.

First claim

1 . A machine-learning process for training and sharing generic functional modules across multiple diverse (architecture, task) pairs for solving multiple diverse problems, comprising:
decomposing by one or more specially programmed processors each of the multiple (architecture, task) pairs into equally sized pseudo-tasks; aligning by the one or more specially programmed processors pseudo-tasks across the multiple diverse architectures; and sharing by the one or more specially programmed processors learned parameters across the aligned pseudo-tasks, wherein each diverse architecture is preserved in performance of its paired task.

2 . The process according to claim 1 , wherein the diverse architectures include core layers selected from the following group consisting of 2D convolutional, LSTM, 1D convolutional and Dense.

3 . The process according to claim 1 , wherein the pseudo-tasks are solved by the functional modules and aligning pseudo-tasks includes optimizing a mapping between the pseudo-tasks and the functional modules that solve them.

4 . The process according to claim 3 , wherein the optimizing uses a stochastic algorithm.

5 . The process according to claim 1 , wherein the sharing by the one or more specially programmed processors learned parameters across the aligned pseudo-tasks is implemented using factorization.

6 . The process according to claim 1 , wherein the multiple diverse problems are selected from the group consisting of a vision problem, a sorting problem, a natural language processing problem, a speech problem, a biological problem, a geological problem and an astronomical problem.

7 . A machine-learning process for training and sharing functional modules across diverse architectures for performing diverse tasks without changing functional forms of underlying predictive models, comprising:
decomposing by one or more specially programmed processors each parameter set for each predictive model into parameter blocks, wherein a parameter block is parameterized by a module; sharing by the one or more specially programmed processors modules across the diverse architectures in accordance with a predetermined alignment, wherein the diverse architectures perform diverse tasks and the sharing of modules improves performance in each diverse task.

8 . The process according to claim 7 , wherein the diverse architectures include core layers selected from the following group consisting of 2D convolutional, LSTM, 1D convolutional and Dense.

9 . The process according to claim 7 , wherein the functional modules solve pseudo-tasks in accordance with associated parameter sets and the predetermined alignment includes aligning by the one or more specially programmed processors, pseudo-tasks across the multiple diverse architectures.

10 . The process according to claim 9 , wherein the predetermined alignment further includes optimizing a mapping between the pseudo-tasks and the functional modules that solve them.

11 . The process according to claim 10 , wherein the optimizing uses a stochastic algorithm.

12 . The process according to claim 9 , wherein the sharing by the one or more specially programmed processors includes sharing learned parameters across the aligned pseudo-tasks.

13 . The process according to claim 12 , wherein sharing learned parameters across the aligned pseudo-tasks is implemented using factorization.

14 . The process according to claim 7 , wherein the multiple diverse tasks are selected from the group consisting of a vision-related task, a sorting-related task, a natural language processing-related task, a speech-related task, a biological-related task, a geological-related task and an astronomical-related task.

15 . A computer-implemented learning process for training and sharing generic functional modules across multiple diverse (architecture, task) pairs for solving multiple diverse problems, comprising:
means for decomposing by one or more specially programmed processors each of the multiple (architecture, task) pairs into equally sized pseudo-tasks; means for aligning by the one or more specially programmed processors pseudo-tasks across the multiple diverse architectures; and means for sharing by the one or more specially programmed processors learned parameters across the aligned pseudo-tasks, wherein each diverse architecture is preserved in performance of its paired task.

16 . The process according to claim 15 , further comprising: means for optimizing a mapping between the pseudo-tasks and the functional modules that solve them.

17 . The process according to claim 16 , wherein the means is a stochastic algorithm.

18 . The process according to claim 17 , wherein the means for sharing by the one or more specially programmed processors learned parameters across the aligned pseudo-tasks includes factorization.

Assignees

• Cognizant Tech Solutions U S Corporation

Inventors

• Meyerson Elliot
• Miikkulainen Risto

Classifications

• G06N3/047 Primary

  Probabilistic or stochastic networks · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title