Guided training of machine learning models with convolution layer feature data fusion

Patent metadata

Field	Value
Publication number	US-12236349-B2
Application number	US-202017098159-A
Country	US
Kind code	B2
Filing date	Nov 13, 2020
Priority date	Nov 14, 2019
Publication date	Feb 25, 2025
Grant date	Feb 25, 2025

Abstract

Aspects described herein provide a method of performing guided training of a neural network model, including: receiving supplementary domain feature data; providing the supplementary domain feature data to a fully connected layer of a neural network model; receiving from the fully connected layer supplementary domain feature scaling data; providing the supplementary domain feature scaling data to an activation function; receiving from the activation function supplementary domain feature weight data; receiving a set of feature maps from a first convolution layer of the neural network model; fusing the supplementary domain feature weight data with the set of feature maps to form fused feature maps; and providing the fused feature maps to a second convolution layer of the neural network model.

First claim

What is claimed is:

1. A method, comprising:
receiving a primary domain feature map from a first layer of a neural network model;
receiving supplementary domain feature data;
generating a supplementary domain feature map based on scaled supplementary domain feature data, wherein the supplementary domain feature map is normalized based on supplementary domain feature scaling data, and a fully connected layer of the neural network model is configured to scale the supplementary domain feature data from a first dimensionality to a second dimensionality associated with an output of a pooling layer of the neural network model;
fusing the supplementary domain feature map with the primary domain feature map to generate a fused feature map; and
providing the fused feature map to a second layer of the neural network model.

2. The method of claim 1 , wherein generating the supplementary domain feature map based on the scaled supplementary domain feature data comprises:
providing the supplementary domain feature data to the fully connected layer;
receiving from the fully connected layer the supplementary domain feature scaling data;
providing the supplementary domain feature scaling data to an activation function for scaling weights associated with individual supplementary domain features; and
receiving from the activation function the supplementary domain feature map normalized based on the scaled weights.

3. The method of claim 2 , wherein the activation function is a non-linear activation function.

4. The method of claim 3 , wherein the non-linear activation function is a sigmoid function.

5. The method of claim 1 , wherein fusing the supplementary domain feature map with the primary domain feature map comprises performing an element-wise multiplication between the supplementary domain feature map and the primary domain feature map.

6. The method of claim 1 , wherein:
the first layer comprises a first convolution layer, and
the second layer comprises a second convolution layer.

7. The method of claim 1 , wherein:
the first layer comprises a pooling layer, and
the second layer comprises a convolution layer.

8. The method of claim 1 , wherein:
the supplementary domain feature data comprises supplementary image features, and
the primary domain feature map comprises image data.

9. The method of claim 1 , further comprising training the neural network model based at least in part on the fused feature map.

10. A processing system configured to performing guided training of a neural network model, comprising:
at least one memory comprising computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions and cause the processing system to:
receive a primary domain feature map from a first layer of a neural network model;
receive supplementary domain feature data;
generate a supplementary domain feature map based on scaled supplementary domain feature data, wherein the supplementary domain feature map is normalized based on supplementary domain feature scaling data, and a fully connected layer of the neural network model is configured to scale the supplementary domain feature data from a first dimensionality to a second dimensionality associated with an output of a pooling layer of the neural network model;
fuse the supplementary domain feature map with the primary domain feature map to generate a fused feature map; and
provide the fused feature map to a second layer of the neural network model.

11. The processing system of claim 10 , wherein in order to generate the supplementary domain feature map based on the scaled supplementary domain feature data, the one or more processors are further configured to cause the processing system to:
provide the supplementary domain feature data to the fully connected layer;
receive from the fully connected layer the supplementary domain feature scaling data;
provide the supplementary domain feature scaling data to an activation function for scaling weights associated with individual supplementary domain features; and
receive from the activation function the supplementary domain feature map normalized based on the scaled weights.

12. The processing system of claim 11 , wherein the activation function is a non-linear activation function.

13. The processing system of claim 12 , wherein the non-linear activation function is a sigmoid function.

14. The processing system of claim 10 , wherein in order to fuse the supplementary domain feature map with the primary domain feature map, the one or more processors are further configured to cause the processing system to perform an element-wise multiplication between the supplementary domain feature map and the primary domain feature map.

15. The processing system of claim 10 , wherein:
the first layer comprises a first convolution layer, and
the second layer comprises a second convolution layer.

16. The processing system of claim 10 , wherein:
the first layer comprises a pooling layer, and
the second layer comprises a convolution layer.

17. The processing system of claim 10 , wherein:
the supplementary domain feature data comprises supplementary image features, and
the primary domain feature map comprises image data.

18. The processing system of claim 10 , wherein the one or more processors are further configured to cause the processing system to train the neural network model based at least in part on the fused feature map.

19. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method, the method comprising:
receiving a primary domain feature map from a first layer of a neural network model;
receiving supplementary domain feature data;
generating a supplementary domain feature map based on scaled supplementary domain feature data, wherein the supplementary domain feature map is normalized based on supplementary domain feature scaling data, and a fully connected layer of the neural network model is configured to scale the supplementary domain feature data from a first dimensionality to a second dimensionality associated with an output of a pooling layer of the neural network model;
fusing the supplementary domain feature map with the primary domain feature map to generate a fused feature map; and
providing the fused feature map to a second layer of the neural network model.

20. The non-transitory computer-readable medium of claim 19 , wherein generating the supplementary domain feature map based on the scaled supplementary domain feature data comprises:
providing the supplementary domain feature data to the fully connected layer;
receiving from the fully connected layer the supplementary domain feature scaling data;
providing the supplementary domain feature scaling data to an activation function for scaling weights associated with individual supplementary domain features; and
receiving from the activation function the supplementary domain feature map normalized based on the scaled weights.

21. The non-transitory computer-readable medium of claim 20 , wherein the activation function is a sigmoid function.

22. The non-transitory computer-readable medium of claim 19 , wherein fusing the supplementary domain feature map with the primary domain feature map comprises performing an element-wise multiplication between the supplementary domain feature map and the primary domain featu

Assignees

• Qualcomm Inc

Inventors

• Borse Shubhankar Mange
• Kwak Nojun
• DIJKMAN Daniel Hendricus Franciscus
• Major Bence

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N3/048

  Activation functions · CPC title