Multi-module and multi-task machine learning system based on an ensemble of datasets

Patent metadata

Field	Value
Publication number	US-11960843-B2
Application number	US-201916401548-A
Country	US
Kind code	B2
Filing date	May 2, 2019
Priority date	May 2, 2019
Publication date	Apr 16, 2024
Grant date	Apr 16, 2024

Abstract

Techniques and systems are provided for training a machine learning model using different datasets to perform one or more tasks. The machine learning model can include a first sub-module configured to perform a first task and a second sub-module configured to perform a second task. The first sub-module can be selected for training using a first training dataset based on a format of the first training dataset. The first sub-module can then be trained using the first training dataset to perform the first task. The second sub-module can be selected for training using a second training dataset based on a format of the second training dataset. The second sub-module can then be trained using the second training dataset to perform the second task.

First claim

What is claimed is:

1. A method of training a machine learning model, comprising:
generating a neural network model to perform a referring expression task based on a plurality of training datasets, wherein each training dataset includes data of a different data format, wherein the neural network model includes an image classification sub-module configured to perform an image classification task and a phrase parsing sub-module configured to perform a phrase parsing task, wherein the image classification task and phrase parsing task are combined to obtain the referring expression task of the neural network model;
receiving the plurality of training datasets, wherein the plurality of training datasets comprises a first training dataset comprising images and a second training dataset comprising natural language phrases, wherein the first training dataset is an image classification dataset, and wherein the second training dataset is a natural language dataset;
determining a data format associated with the first training dataset is an image format;
identifying, using a lookup table, the image classification sub-module as being associated with the image format;
training the image classification sub-module using the images of the image classification dataset, wherein the image classification sub-module is trained to perform the image classification task;
determining a data format associated with the second training dataset is a text format identifying, using the lookup table, the phrase parsing sub-module as being associated with the text format; and
training the phrase parsing sub-module using the natural language phrases of the natural language dataset, wherein the phrase parsing sub-module is trained to perform the phrase parsing task.

2. The method of claim 1 , further comprising:
obtaining an output from the image classification sub-module based on the training of the image classification sub-module using the image classification dataset; and
selecting an additional dataset for training the image classification sub-module based on the obtained output.

3. The method of claim 1 , wherein a combination of a portion of the image classification dataset and a portion of the natural language dataset is processed by the neural network model.

4. The method of claim 3 , wherein a percentage of data from the image classification dataset and a percentage of data from the natural language dataset included in the combination are configurable using one or more parameters input to the neural network model.

5. The method of claim 1 , wherein the neural network model includes at least one shared layer included in the image classification sub-module and the phrase parsing sub-module, and includes at least one non-shared layer included in the image classification sub-module and not included in the phrase parsing sub-module.

6. The method of claim 5 , wherein the at least one shared layer is trained using the image classification dataset and the natural language dataset, and wherein the at least one non-shared layer is trained using the image classification dataset.

7. The method of claim 1 , further comprising:
obtaining a training dataset; and
training, using the training dataset, the image classification sub-module and the phrase parsing sub-module to perform at least another task.

8. The method of claim 1 , further comprising:
obtaining a training dataset; and
training, using the image classification dataset and the training dataset, the image classification sub-module to perform at least the image classification task.

9. A machine learning system, comprising:
an input device configured to receive a plurality of training datasets comprising an image classification dataset and a natural language dataset;
a neural network model for performing a referring expression task, including an image processing sub-module with a first plurality of neural network layers for performing an object detection task and a natural language sub-module with a second plurality of neural network layers for performing a phrase parsing task, wherein the object detection task and phrase parsing task are combined to obtain the referring expression task of the neural network model;
a sub-module determination engine configured to:
receive the plurality of training datasets, wherein the plurality of training datasets comprises a first training dataset comprising images and a second training dataset comprising natural language phrases;
determine a data format associated with the first training dataset is an object detection task;
determine a data format associated with the second training dataset is a phrase parsing task;
identifying, using a lookup table, the image processing sub-module as being associated with the object detection task;
identifying, using the lookup table, the natural language sub-module as being associated with the phrase parsing task;
training the first plurality of neural network layers of the image processing sub-module using the image classification dataset, wherein the image processing sub-module is trained to perform the object detection task; and
training the second plurality of neural network layers of the natural language sub-module using the natural language dataset, wherein the natural language sub-module is trained to perform the phrase parsing task; and

an output device configured to output the referring expression task of the neural network model based on the image processing sub-module or the natural language sub-module.

10. The machine learning system of claim 9 , wherein the output device is configured to obtain an output from the image processing sub-module based on the training of the image processing sub-module using the image classification dataset, and wherein the input device is configured to select an additional dataset for training the image processing sub-module based on the obtained output.

11. The machine learning system of claim 9 , wherein the input device is configured to receive one or more parameters, and wherein the neural network model is configured to use the one or more parameters to determine a percentage of data from the image classification dataset and a percentage of data from the natural language dataset to use for training the image processing sub-module and the natural language sub-module.

12. The machine learning system of claim 9 , wherein the image processing sub-module and the natural language sub-module include at least one shared neural network layer included in both the image processing sub-module and the natural language sub-module, and wherein the image processing sub-module includes at least one non-shared neural network layer included in the image processing sub-module and not included in the natural language sub-module.

13. The machine learning system of claim 12 , wherein the at least one shared neural network layer is trained using the image classification dataset and the natural language dataset, and wherein the at least one non-shared neural network layer is trained using the image classification dataset and not the natural language dataset.

14. The machine learning system of claim 9 , wherein the input device is configured to obtain a training dataset, and wherein the first plurality of neural network layers of the image processing sub-module and the second plurality of neural network layers of the natural language sub-module are trained using the training dataset to perform at least another task.

15. The method of claim 1 , wherein training, using the image classification dataset, the image classification sub-module of the neural network model and training, using the

Assignees

• Adobe Inc

Inventors

• Lin Zhe
• Bui Trung Huu
• Cohen Scott
• Ling Mingyang
• Wu Chenyun

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title