Visual-semantic representation learning via multi-modal contrastive training

Patent metadata

Field	Value
Publication number	US-12223439-B2
Application number	US-202117190668-A
Country	US
Kind code	B2
Filing date	Mar 3, 2021
Priority date	Mar 3, 2021
Publication date	Feb 11, 2025
Grant date	Feb 11, 2025

Abstract

Systems and methods for multi-modal representation learning are described. One or more embodiments provide a visual representation learning system trained using machine learning techniques. For example, some embodiments of the visual representation learning system are trained using cross-modal training tasks including a combination of intra-modal and inter-modal similarity preservation objectives. In some examples, the training tasks are based on contrastive learning techniques.

First claim

The invention claimed is:

1. A method of training a machine learning model, the method comprising:
identifying a training set comprising a plurality of images and a plurality of captions corresponding to the images;
encoding the images using an image encoder to produce encoded images;
encoding the captions using a text encoder to produce encoded text;
computing a multi-modal loss function based on the encoded images and the encoded text, the multi-modal loss function comprising at least one image loss term, at least one text loss term, and at least one cross-modal term; and
training the image encoder and the text encoder based on the multi-modal loss function.

2. The method of claim 1 , further comprising:
computing an image self-supervised contrastive loss, wherein the at least one image loss term includes the image self-supervised contrastive loss.

3. The method of claim 1 , further comprising:
computing a tag-supervised contrastive loss, wherein the at least one image loss term includes the image tag-supervised contrastive loss.

4. The method of claim 1 , further comprising:
computing a caption self-supervised contrastive loss, wherein the at least one text loss term includes the caption self-supervised contrastive loss.

5. The method of claim 1 , further comprising:
computing a caption-image contrastive loss, wherein the at least one cross-modal term includes the caption-image contrastive loss.

6. The method of claim 1 , further comprising:
computing an image-caption contrastive loss, wherein the at least one cross-modal term includes the image-caption contrastive loss.

7. The method of claim 1 , further comprising:
encoding the images using a momentum image encoder to produce momentum encoded images, wherein the at least one image loss term is based on the encoded images and the momentum encoded images.

8. The method of claim 7 , wherein:
the at least one cross-modal term is based on the momentum encoded images and the encoded text.

9. The method of claim 1 , further comprising:
encoding the captions using a momentum text encoder to produce momentum encoded text, wherein the at least one text loss term is based on the encoded text and the momentum encoded text.

10. The method of claim 9 , wherein:
the at least one cross-modal term is based on the encoded images and the momentum encoded text.

11. The method of claim 1 , wherein:
the multi-modal loss function is based on a contrastive learning framework.

12. The method of claim 1 , wherein:
the encoded images and the encoded text are represented in a same embedding space.

13. The method of claim 1 , further comprising:
adjusting one or more of the images to produce an augmented training set, wherein the training is based on the augmented training set.

14. An apparatus comprising:
an image encoder configured to encode images to produce encoded images;
a text encoder configured to encode captions corresponding to the images to produce encoded text; and
a training component configured to compute a multi-modal loss function based on the encoded images and the encoded text and to train the image encoder and the text encoder based on the multi-modal loss function, wherein the multi-modal loss function comprises at least one image loss term, at least one text loss term, and at least one cross-modal term.

15. The apparatus of claim 14 , further comprising:
a momentum image encoder configured to encode the images to produce momentum encoded images, wherein the at least one image loss term is based on the encoded images and the momentum encoded images.

16. The apparatus of claim 14 , further comprising:
a momentum text encoder configured to encode the captions to produce momentum encoded text, wherein the at least one text loss term is based on the encode text and the momentum encoded text.

17. The apparatus of claim 14 , wherein:
the image encoder comprises a first image output head and a second image output head, wherein the at least one image loss term is based on the first image output head and the at least one cross-modal term is based on the second image output head.

18. The apparatus of claim 14 , wherein:
the text encoder comprises a first text output head and a second text output head, wherein the at least one text loss term is based on the first text output head and the at least one cross-modal term is based on the second text output head.

19. A method of image search comprising:
encoding an image using an image encoder to produce an encoded image;
encoding text using a text encoder to produce encoded text, wherein the image encoder and the text encoder are jointly trained based on a multi-modal loss function comprising at least one image loss term, at least one text loss term, and at least one cross-modal term; and
performing an image search based on the encoded image and the encoded text.

20. The method of claim 19 , wherein: the image search comprises retrieving search text corresponding to a query image, retrieving a search image corresponding to a query text, or retrieving the search image corresponding to the query image.

Assignees

• Adobe Inc

Inventors

• Yuan Xin
• Lin Zhe
• Kuen Jason Wen Yong
• Zhang Jianming
• Wang Yilin
• Kale Ajinkya
• Faieta Baldo

Classifications

• G06N3/096

  Transfer learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06T2207/00

  Indexing scheme for image analysis or image enhancement · CPC title