Continual learning using cross connections

Patent metadata

Field	Value
Publication number	US-12327184-B2
Application number	US-202016923196-A
Country	US
Kind code	B2
Filing date	Jul 8, 2020
Priority date	Jul 8, 2020
Publication date	Jun 10, 2025
Grant date	Jun 10, 2025

Abstract

A method, a computer system, and a computer program product for using distinct paths with cross connections for distinct tasks to prevent catastrophic forgetting in class-incremental scenarios. Embodiments of the present invention may include receiving one or more tasks sequentially. Embodiments of the present invention may include applying one or more shareable blocks to the one or more tasks. Embodiments of the present invention may include learning one or more distinct paths for the one or more tasks. Embodiments of the present invention may include adding one or more cross connections between the one or more tasks. Embodiments of the present invention may include adding an aggregation block to collect one or more outputs from the distinct paths of each of the one or more tasks. Embodiments of the present invention may include providing a prediction.

First claim

What is claimed is:

1. A method comprising:
receiving one or more tasks sequentially in a deep neural network architecture;
applying one or more shareable blocks in a bottom layer of the deep neural network architecture to the one or more tasks;
freezing one or more neurons of one or more previously learned distinct paths of the one or more tasks, wherein freezing comprises setting a weight associated with one or more parameters of a neuron as untrainable;
training the deep neural network through:
identifying one or more distinct paths for the one or more tasks;
identifying hidden activations, sequentially along the one or more distinct paths, for each task, wherein the hidden activations are outputs of each task and are calculated as h 1 (i) =B 1 (i) (h 1 (i−1) ), i∈{1, 2, . . . , m} where B is a current block and h is input data sampled from a training dataset;
creating one or more cross connections between the one or more tasks and one or more previous tasks, and wherein the one or more cross connections are selected from a group consisting of a forward connection from the one or more previous tasks to the one or more tasks and a backward connection from the one or more tasks to the one or more previous tasks; and
aggregating the outputs from the distinct paths of each of the one or more tasks in an aggregation block; and

providing a prediction as to a task in the one or more tasks using the trained deep neural network.

2. The method of claim 1 , wherein receiving the one or more tasks sequentially further comprises:
receiving one or more labels of data, wherein the one or more labels of data correspond to the one or more tasks.

3. The method of claim 1 , further comprising:
training one or more neurons of the one or more distinct paths for the one or more tasks, wherein the one or more neurons are set as trainable.

4. The method of claim 3 , wherein freezing the one or more neurons of one or more previously learned distinct paths of the one or more tasks further comprises:
setting the one or more neurons of the one or more previously learned distinct paths as untrainable.

5. The method of claim 1 , wherein the one or more distinct paths comprises one or more neurons, wherein the one or more neurons have one or more weights.

6. The method of claim 1 , wherein the one or more cross connections between the one or more tasks comprises:
one or more forward cross connections between the one or more tasks; and
one or more backward cross connections between the one or more tasks.

7. The method of claim 1 , wherein collecting the outputs from the distinct paths of each of the one or more tasks further comprises:
aggregating, by the aggregation block, one or more hidden activations from the one or more distinct paths of the one or more tasks.

8. The method of claim 1 , wherein the aggregation block is added before a final layer of a deep learning network.

9. A computer system, comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising:
receiving one or more tasks sequentially in a deep neural network architecture;
applying one or more shareable blocks in a bottom layer of the deep neural network architecture to the one or more tasks;
freezing one or more neurons of one or more previously learned distinct paths of the one or more tasks, wherein freezing comprises setting a weight associated with one or more parameters of a neuron as untrainable;
training the deep neural network through:
identifying one or more distinct paths for the one or more tasks;
identifying hidden activations, sequentially along the one or more distinct paths, for each task, wherein the hidden activations are outputs of each task and are calculated as h 1 (i) =B 1 (i) (h 1 (i−1) ), i∈{1, 2, . . . , m} where B is a current block and h is input data sampled from a training dataset;
creating one or more cross connections between the one or more tasks and one or more previous tasks, and wherein the one or more cross connections are selected from a group consisting of a forward connection from the one or more previous tasks to the one or more tasks and a backward connection from the one or more tasks to the one or more previous tasks; and
aggregating the outputs from the distinct paths of each of the one or more tasks in an aggregation block; and

providing a prediction as to a task in the one or more tasks using the trained deep neural network.

10. The computer system of claim 9 , wherein receiving the one or more tasks sequentially further comprises:
receiving one or more labels of data, wherein the one or more labels of data correspond to the one or more tasks.

11. The computer system of claim 9 , further comprising:
training one or more neurons of the one or more distinct paths for the one or more tasks, wherein the one or more neurons are set as trainable.

12. The computer system of claim 11 , wherein freezing the one or more neurons of one or more previously learned distinct paths of the one or more tasks further comprises:
setting the one or more neurons of the one or more previously learned distinct paths as untrainable.

13. The computer system of claim 9 , wherein the one or more distinct paths comprises one or more neurons, wherein the one or more neurons have one or more weights.

14. The computer system of claim 9 , wherein the one or more cross connections between the one or more tasks comprises:
one or more forward cross connections between the one or more tasks; and
one or more backward cross connections between the one or more tasks.

15. The computer system of claim 9 , wherein collecting the outputs from the distinct paths of each of the one or more tasks further comprises:
aggregating, by the aggregation block, one or more hidden activations from the one or more distinct paths of the one or more tasks.

16. The computer system of claim 9 , wherein the aggregation block is added before a final layer of a deep learning network.

17. A computer program product, comprising:
one or more computer-readable tangible storage media and program instructions stored on at least one of the one or more computer-readable tangible storage media, the program instructions executable by a processor to cause the processor to perform a method comprising:
receiving one or more tasks sequentially in a deep neural network architecture;
applying one or more shareable blocks in a bottom layer of the deep neural network architecture to the one or more tasks;
freezing one or more neurons of one or more previously learned distinct paths of the one or more tasks, wherein freezing comprises setting a weight associated with one or more parameters of a neuron as untrainable;
training the deep neural network through:
identifying one or more distinct paths for the one or more tasks;
identifying hidden activations, sequentially along the one or more distinct paths, for each task, wherein the hidden activations are outputs of each task and are calculated as h 1 (i) =B 1 (i) (h 1 (i−1) ), i∈{1, 2, . . . , m} where B is a current block and h is input data sampled from a training dataset;
creating one or more cross connections between the one or more tasks and one or more previous tasks, and wherein

Assignees

• IBM

Inventors

• Tian Yu
• Wang Xiaolong
• Xue Qinghan
• Eliuk Steven Nicholas
• Guo Xin

Classifications

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title