Continual learning using cross connections

Patent metadata

Field	Value
Publication number	US-2022012583-A1
Application number	US-202016923196-A
Country	US
Kind code	A1
Filing date	Jul 8, 2020
Priority date	Jul 8, 2020
Publication date	Jan 13, 2022
Grant date	—

Abstract

A method, a computer system, and a computer program product for using distinct paths with cross connections for distinct tasks to prevent catastrophic forgetting in class-incremental scenarios. Embodiments of the present invention may include receiving one or more tasks sequentially. Embodiments of the present invention may include applying one or more shareable blocks to the one or more tasks. Embodiments of the present invention may include learning one or more distinct paths for the one or more tasks. Embodiments of the present invention may include adding one or more cross connections between the one or more tasks. Embodiments of the present invention may include adding an aggregation block to collect one or more outputs from the distinct paths of each of the one or more tasks. Embodiments of the present invention may include providing a prediction.

First claim

What is claimed is:

1 . A method comprising:
receiving one or more tasks sequentially; applying one or more shareable blocks to the one or more tasks; learning one or more distinct paths for the one or more tasks; adding one or more cross connections between the one or more tasks; adding an aggregation block to collect one or more outputs from the distinct paths of each of the one or more tasks; and providing a prediction.

2 . The method of claim 1 , wherein receiving the one or more tasks sequentially further comprises:
receiving one or more labels of data, wherein the one or more labels of data correspond to the one or more tasks.

3 . The method of claim 1 , further comprising:
freezing one or more neurons of one or more previously learned distinct paths of the one or more tasks; and training one or more neurons of the one or more distinct paths for the one or more tasks, wherein the one or more neurons are set as trainable.

4 . The method of claim 3 , wherein freezing the one or more neurons of one or more previously learned distinct paths of the one or more tasks further comprises:
setting the one or more neurons of the one or more previously learned distinct paths as untrainable.

5 . The method of claim 1 , wherein the one or more distinct paths comprises one or more neurons, wherein the one or more neurons have one or more weights.

6 . The method of claim 1 , wherein the one or more cross connections between the one or more tasks comprises:
one or more forward cross connections between the one or more tasks; and one or more backward cross connections between the one or more tasks.

7 . The method of claim 1 , wherein collecting the one or more outputs from the distinct paths of each of the one or more tasks further comprises:
aggregating, by the aggregation block, one or more hidden activations from the one or more distinct paths of the one or more tasks.

8 . The method of claim 1 , wherein the aggregation block is added before a final layer of a deep learning network.

9 . A computer system, comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising:
receiving one or more tasks sequentially;
applying one or more shareable blocks to the one or more tasks;
learning one or more distinct paths for the one or more tasks;
adding one or more cross connections between the one or more tasks;
adding an aggregation block to collect one or more outputs from the distinct paths of each of the one or more tasks; and
providing a prediction.

10 . The computer system of claim 9 , wherein receiving the one or more tasks sequentially further comprises:
receiving one or more labels of data, wherein the one or more labels of data correspond to the one or more tasks.

11 . The computer system of claim 9 , further comprising:
freezing one or more neurons of one or more previously learned distinct paths of the one or more tasks; and training one or more neurons of the one or more distinct paths for the one or more tasks, wherein the one or more neurons are set as trainable.

12 . The computer system of claim 11 , wherein freezing the one or more neurons of one or more previously learned distinct paths of the one or more tasks further comprises:
setting the one or more neurons of the one or more previously learned distinct paths as untrainable.

13 . The computer system of claim 9 , wherein the one or more distinct paths comprises one or more neurons, wherein the one or more neurons have one or more weights.

14 . The computer system of claim 9 , wherein the one or more cross connections between the one or more tasks comprises:
one or more forward cross connections between the one or more tasks; and one or more backward cross connections between the one or more tasks.

15 . The computer system of claim 9 , wherein collecting the one or more outputs from the distinct paths of each of the one or more tasks further comprises:
aggregating, by the aggregation block, one or more hidden activations from the one or more distinct paths of the one or more tasks.

16 . The computer system of claim 9 , wherein the aggregation block is added before a final layer of a deep learning network.

17 . A computer program product, comprising:
one or more computer-readable tangible storage media and program instructions stored on at least one of the one or more computer-readable tangible storage media, the program instructions executable by a processor to cause the processor to perform a method comprising:
receiving one or more tasks sequentially;
applying one or more shareable blocks to the one or more tasks;
learning one or more distinct paths for the one or more tasks;
adding one or more cross connections between the one or more tasks;
adding an aggregation block to collect one or more outputs from the distinct paths of each of the one or more tasks; and
providing a prediction.

18 . The computer program product of claim 17 , wherein receiving the one or more tasks sequentially further comprises:
receiving one or more labels of data, wherein the one or more labels of data correspond to the one or more tasks.

19 . The computer program product of claim 17 , further comprising:
freezing one or more neurons of one or more previously learned distinct paths of the one or more tasks; and training one or more neurons of the one or more distinct paths for the one or more tasks, wherein the one or more neurons are set as trainable.

20 . The computer program product of claim 19 , wherein freezing the one or more neurons of one or more previously learned distinct paths of the one or more tasks further comprises:
setting the one or more neurons of the one or more previously learned distinct paths as untrainable.

21 . The computer program product of claim 17 , wherein the one or more distinct paths comprises one or more neurons, wherein the one or more neurons have one or more weights.

22 . The computer program product of claim 17 , wherein the one or more cross connections between the one or more tasks comprises:
one or more forward cross connections between the one or more tasks; and one or more backward cross connections between the one or more tasks.

23 . The computer program product of claim 17 , wherein collecting the one or more outputs from the distinct paths of each of the one or more tasks further comprises:
aggregating, by the aggregation block, one or more hidden activations from the one or more distinct paths of the one or more tasks.

24 . The computer program product of claim 17 wherein the aggregation block is added before a final layer of a deep learning network.

Assignees

• IBM

Inventors

• Tian Yu
• Wang Xiaolong
• Xue Qinghan
• Eliuk Steven Nicholas
• Guo Xin

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044 Primary

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/09

  Supervised learning · CPC title