Joint training of neural networks using multi-scale hard example mining

Patent metadata

Field	Value
Publication number	US-11790631-B2
Application number	US-202117408094-A
Country	US
Kind code	B2
Filing date	Aug 20, 2021
Priority date	Apr 7, 2017
Publication date	Oct 17, 2023
Grant date	Oct 17, 2023

Abstract

An example apparatus for mining multi-scale hard examples includes a convolutional neural network to receive a mini-batch of sample candidates and generate basic feature maps. The apparatus also includes a feature extractor and combiner to generate concatenated feature maps based on the basic feature maps and extract the concatenated feature maps for each of a plurality of received candidate boxes. The apparatus further includes a sample scorer and miner to score the candidate samples with multi-task loss scores and select candidate samples with multi-task loss scores exceeding a threshold score.

First claim

What is claimed is:

1. An apparatus for mining multi-scale hard examples comprising:
memory;
machine readable instructions; and
processor circuitry to execute the machine readable instructions to at least:
generate candidate feature maps based on a mini-batch of sample candidates;
enable multi-scale extraction functionality by generating concatenated feature maps based on the candidate feature maps;
improve feature discrimination by extracting the concatenated feature maps for each of a plurality of received candidate boxes at two or more scales;
train a convolutional neural network with the concatenated feature maps;
improve accuracy of the trained convolutional neural network by scoring the candidate samples with multi-task loss scores, the multi-task loss scores corresponding to loss information; and
in response to detecting multi-task loss scores satisfying a threshold value, cause respective ones of the candidate samples to be selected for training.

2. The apparatus of claim 1 , wherein the loss information includes at least one of classification loss information or localization loss information.

3. The apparatus of claim 1 , wherein the processor circuitry is to generate candidate boxes by instantiating a region proposal network in response to receiving a plurality of mini-batches of images.

4. The apparatus of claim 1 , wherein respective ones of the concatenated feature maps include a plurality of channels having the candidate feature maps resized to a reference layer size.

5. The apparatus of claim 1 , wherein the multi-task loss score for each candidate sample is to be calculated based on a localization score and a classification score corresponding to classification and localization losses calculated for each candidate sample in a respective Stochastic Gradient Descent (SGD).

6. The apparatus of claim 1 , wherein the processor circuitry is to invoke the convolutional neural network to resize sample candidates from the mini-batch into a standard scale.

7. The apparatus of claim 1 , wherein the processor circuitry is to generate the concatenated feature maps by:
selecting a reference layer in the convolutional neural network; and
up-sampling or down-sampling feature maps from other layers in the convolutional neural network, respective ones of the concatenated feature maps including a feature map of the reference layer and the up-sampled or down-sampled feature maps of other layers in the convolutional neural network.

8. A computer readable storage device or storage disk comprising instructions that, when executed, cause processor circuitry to at least:
generate candidate feature maps based on a mini-batch of sample candidates;
enable multi-scale extraction functionality by generating concatenated feature maps based on the candidate feature maps;
improve feature discrimination by extracting the concatenated feature maps for each of a plurality of received candidate boxes at two or more scales;
train a convolutional neural network with the concatenated feature maps;
improve accuracy of the trained convolutional neural network by scoring the candidate samples with multi-task loss scores, the multi-task loss scores corresponding to loss information; and
in response to detecting multi-task loss scores satisfying a threshold score, cause respective ones of the candidate samples to be selected for training.

9. The storage device or storage disk of claim 8 , wherein the loss information includes at least one of classification loss information or localization loss information.

10. The storage device or storage disk of claim 8 , wherein the instructions, when executed, cause the processor circuitry to generate candidate boxes by instantiating a region proposal network in response to receiving a plurality of mini-batches of images.

11. The storage device or storage disk of claim 8 , wherein respective ones of the concatenated feature maps include a plurality of channels having the candidate feature maps resized to a reference layer size.

12. The storage device or storage disk of claim 8 , wherein the instructions, when executed, cause the processor circuitry to calculate the multi-task loss score for each candidate sample based on a localization score and a classification score corresponding to the classification and localization losses calculated for each candidate sample in a respective Stochastic Gradient Descent (SGD).

13. The storage device or storage disk of claim 8 , wherein the instructions, when executed, cause the processor circuitry to invoke the convolutional neural network to resize sample candidates from the mini-batch into a standard scale.

14. The storage device or storage disk of claim 8 , wherein the instructions, when executed, cause the processor circuitry to generate the concatenated feature maps by:
selecting a reference layer in the convolutional neural network; and
up-sampling or down-sampling feature maps from other layers in the convolutional neural network, respective ones of the concatenated feature maps including a feature map of the reference layer and the up-sampled or down-sampled feature maps of other layers in the convolutional neural network.

15. A system for mining multi-scale hard examples, comprising:
means for generating candidate feature maps using a mini-batch of sample candidates;
means for enabling multi-scale extraction functionality by generating concatenated feature maps based on the candidate feature maps;
means for improving feature discrimination by extracting the concatenated feature maps for each of a plurality of received candidate boxes;
means for training a convolutional neural network with the concatenated feature maps;
means for improving accuracy of the trained convolutional neural network by scoring the candidate samples with multi-task loss scores, the multi-task loss scores corresponding to loss information; and
means for causing respective ones of the candidate samples to be selected for training in response to detecting multi-task scores satisfying a threshold score.

16. The system of claim 15 , wherein the candidate boxes are to be generated by a region proposal network in response to receiving a plurality of mini-batches of example images.

17. The system of claim 15 , wherein each of the concatenated feature maps include a plurality of channels comprising the feature maps resized to a reference layer size.

18. The system of claim 15 , wherein the multi-task loss score for each candidate sample is to be calculated based on a localization score and a classification score corresponding to the classification and localization losses calculated for each candidate sample in a respective Stochastic Gradient Descent (SGD).

19. The system of claim 15 , wherein the means for generating candidate feature maps is to resize sample candidates from the received mini-batch into a standard scale.

20. The system of claim 15 , wherein the concatenated feature maps are calculated by:
selecting a reference layer in the convolutional neural network; and
up-sampling or down-sampling feature maps from other layers in the convolutional neural network, respective ones of the concatenated feature maps including a feature map of the reference layer and the up-sampled or down-sampled feature maps of other layers in the convolutional neural network.

Assignees

• Intel Corp

Inventors

• Yao Anbang
• Ren Yun
• Zhao Hao
• Kong Tao
• Chen Yurong

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06V10/454 Primary

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06F18/24317

  Piecewise classification, i.e. whereby each classification requires several discriminant rules · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title