Image captioning with weakly-supervised attention penalty

Patent metadata

Field	Value
Publication number	US-11775838-B2
Application number	US-202117501199-A
Country	US
Kind code	B2
Filing date	Oct 14, 2021
Priority date	Oct 15, 2018
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

Techniques for training a machine-learning (ML) model for captioning images are disclosed. A plurality of feature vectors and a plurality of visual attention maps are generated by a visual model of the ML model based on an input image. Each of the plurality of feature vectors correspond to different regions of the input image. A plurality of caption attention maps are generated by an attention model of the ML model based on the plurality of feature vectors. An attention penalty is calculated based on a comparison between the caption attention maps and the visual attention maps. A loss function is calculated based on the attention penalty. One or both of the visual model and the attention model are trained using the loss function.

First claim

What is claimed is:

1. A computer-implemented method of training a machine learning model for captioning images, the computer-implemented method comprising:
receiving an input image;
generating, using a visual model and based on the input image, a feature vector and a visual attention map;
generating, using an attention model and based on the feature vector, a caption attention map;
generating, using a language model and based on one or both of the feature vector and the caption attention map, a predicted caption for the input image;
calculating (i) a caption loss based on a comparison between the predicted caption and a manual caption label and (ii) an attention penalty based on a comparison between the caption attention map and the visual attention map; and
calculating a loss function based on the caption loss and the attention penalty.

2. The computer-implemented method of claim 1 , wherein one or more of the visual model, the attention model, and the language model is trained by:
modifying a set of weights of one or more of the visual model, the attention model, and the language model proportional to a magnitude of the loss function.

3. The computer-implemented method of claim 1 , further comprising:
parsing the predicted caption to identify one or more words related to visual concepts using a category mapping dictionary.

4. The computer-implemented method of claim 3 , wherein the visual model is a pre-trained convolutional neural network.

5. The computer-implemented method of claim 3 , wherein the visual model is fine-tuned using the visual concepts.

6. The computer-implemented method of claim 5 , wherein one or more different attention maps are generated for one or more different visual concepts in the predicted caption.

7. The computer-implemented method of claim 1 , wherein the language model is a long short-term memory model.

8. The computer-implemented method of claim 1 , wherein the attention model comprises one or more of a long short-term memory model and a multi-layer perceptron.

9. A hardware storage device having stored thereon computer-executable instructions that, when executed by one or more processors, configure a computer system to perform at least the following:
receive an input image;
generate, using a visual model and based on the input image, a feature vector and a visual attention map;
generate, using an attention model and based on the feature vector, a caption attention map;
generate, using a language model and based on one or both of the feature vector and the caption attention map, a predicted caption for the input image;
calculate (i) a caption loss based on a comparison between the predicted caption and a manual caption label and (ii) an attention penalty based on a comparison between the caption attention map and the visual attention map; and
calculate a loss function based on the caption loss and the attention penalty.

10. The hardware storage device of claim 9 , wherein the computer-executable instructions further configure the computer system to:
train one or more of the visual model, the attention model, and the language model by modifying a set of weights of one or more of the visual model, the attention model, and the language model proportional to a magnitude of the loss function.

11. The hardware storage device of claim 10 , wherein the visual attention map and the caption attention map are probability distributions.

12. The hardware storage device of claim 9 , wherein a gradients-based weakly supervised model and bottom-up and top-down attention mechanism are used to generate the visual attention map and the caption attention map, respectively.

13. The hardware storage device of claim 9 , wherein the visual model is fine-tuned using a multi-label multi-class classification objective.

14. The hardware storage device of claim 9 , wherein the attention model is a LSTM model configured to generate the caption attention map using the feature vector as inputs.

15. The hardware storage device of claim 9 , wherein the language model is a LSTM model configured to generate the predicted caption using one or more of the feature vector or the caption attention map as inputs.

16. The hardware storage device of claim 9 , wherein for each visual word in the predicted caption, the visual attention map is generated by computing gradients of visual model outputs with respect to a last convolutional layer.

17. A system comprising:
one or more processors; and
one or more hardware storage devices having stored thereon computer-executable instructions that, when executed by the one or more processors, configure the system to perform at least the following:
receive an input image;
generate, using a visual model and based on the input image, a feature vector and a visual attention map;
generate, using an attention model and based on the feature vector, a caption attention map;
generate, using a language model and based on one or both of the feature vector and the caption attention map, a predicted caption for the input image;
calculate (i) a caption loss based on a comparison between the predicted caption and a manual caption label and (ii) an attention penalty based on a comparison between the caption attention map and the visual attention map; and
calculate a loss function based on the caption loss and the attention penalty.

18. The system of claim 17 , wherein the one or more of the visual model, the attention model, and the language model is trained by performing at least the following:
modify a set of weights of one or more of the visual model, the attention model, and the language model proportional to a magnitude of the loss function.

19. The system of claim 18 , wherein the loss function is calculated based on a summation of the caption loss and the attention penalty.

20. The system of claim 17 , wherein the visual attention map is generated by a gradients-based weakly supervised model.

Assignees

• Ancestry Com Operations Inc

Inventors

• Li Jiayun
• Ebrahimpour Mohammad K
• Moghtaderi Azadeh
• Yu Yen-Yun

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title