Localization of narrations in image data

Patent metadata

Field	Value
Publication number	US-12118787-B2
Application number	US-202117499193-A
Country	US
Kind code	B2
Filing date	Oct 12, 2021
Priority date	Oct 12, 2021
Publication date	Oct 15, 2024
Grant date	Oct 15, 2024

Abstract

Methods, system, and computer storage media are provided for multi-modal localization. Input data comprising two modalities, such as image data and corresponding text or audio data, may be received. A phrase may be extracted from the text or audio data, and a neural network system may be utilized to spatially and temporally localize the phrase within the image data. The neural network system may include a plurality of cross-modal attention layers that each compare features across the first and second modalities without comparing features of the same modality. Using the cross-modal attention layers, a region or subset of pixels within one or more frames of the image data may be identified as corresponding to the phrase, and a localization indicator may be presented for display with the image data. Embodiments may also include unsupervised training of the neural network system.

First claim

What is claimed is:

1. A computer-implemented method for localizing language in image data, the computer-implemented method comprising:
receiving input data comprising a video partitioned into a first modality and a second modality, the first modality being image data comprising a plurality of frames and the second modality being one of audio data or text data corresponding to the image data;
extracting a first phrase from the second modality;
identifying, via a neural network system, a first spatiotemporal region in the first modality corresponding to the first phrase, the first spatiotemporal region comprising a first frame from the plurality of frames and a portion of pixels within the first frame, wherein the first spatiotemporal region is identified utilizing a plurality of cross-modal attention layers in the neural network system to compare features from the first modality with features from the second modality; and
causing presentation, on a graphic user interface, of an indicator of the first spatiotemporal region.

2. The computer-implemented method of claim 1 , wherein the plurality of cross-modal attention layers alternate with at least one self-attention layer.

3. The computer-implemented method of claim 1 , wherein a full attention layer is absent from the neural network system.

4. The computer-implemented method of claim 1 , wherein identifying, via the neural network system, the first spatiotemporal region in the first modality corresponding to the first phrase extracted from the second modality comprises:
extracting a set of spatiotemporal features from the first modality and extracting a set of word features from the second modality; and
computing similarity scores between at least the first phrase within the second modality and regions within the first modality using the set of spatiotemporal features and the set of word features, wherein the first spatiotemporal region identified in the first modality as corresponding to the first phrase has the highest similarity score relative to other regions within the first modality.

5. The computer-implemented method of claim 4 , wherein each cross-modal attention layer is bidirectional in that, at each cross-modal attention layer, the set of spatiotemporal features are utilized for a query with the set of word features are utilized for a key and the set of spatiotemporal features are utilized for a key with the set of word features are utilized for a query.

6. The computer-implemented method of claim 1 , wherein the second modality comprises audio data, and wherein extracting the first phrase comprises extracting text from the audio data utilizing automatic speech recognition and performing natural language processing to identify the first phrase.

7. The computer-implemented method of claim 1 , wherein the indicator on the graphic user interface comprises a bounding box around the first spatiotemporal region.

8. A computerized system for localizing language across other modalities, the computerized system comprising:
at least one processor; and
one or more computer storage media storing computer-usable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving input data partitioned into a first modality and a second modality, the first modality being image data comprising a plurality of frames and the second modality being one of audio data or text data corresponding to the image data;
extracting a first phrase from the second modality;
identifying, via a neural network system, a first spatiotemporal region in the first modality corresponding to the first phrase, the first spatiotemporal region comprising a first frame from the plurality of frames and a portion of pixels within the first frame, wherein the first spatiotemporal region is identified utilizing a plurality of cross-modal attention layers in the neural network system to compare features from the first modality with features from the second modality; and
causing presentation, on a graphic user interface, of an indicator of the first spatiotemporal region.

9. The computerized system of claim 8 , wherein the first phrase narrates an interaction between two objects.

10. The computerized system of claim 8 , wherein identifying, via the neural network system, the first spatiotemporal region in the first modality corresponding to the first phrase extracted from the second modality comprises:
extracting a set of spatiotemporal features from the first modality and extracting a set of word features from the second modality; and
computing similarity scores between at least the first phrase within the second modality and multiple regions within the first modality using the set of spatiotemporal features and the set of word features, wherein the first spatiotemporal region identified in the first modality as corresponding to the first phrase has the highest similarity score relative to other regions within the first modality.

11. The computerized system of claim 10 , wherein the operations further comprise, based on similarities scores for regions within the first modality, generating a heat map for the first modality.

12. The computerized system of claim 11 , wherein the indicator of the first spatiotemporal region comprises at least a portion of the heat map corresponding to the first spatiotemporal region.

13. The computerized system of claim 8 , wherein the plurality of cross-modal attention layers alternate with at least one self-attention layer.

14. The computerized system of claim 8 , wherein a full attention layer is absent from the neural network system.

15. The computerized system of claim 8 , wherein the indicator on the graphic user interface comprises a change in one or more of a brightness level and a color channel level for the portion of pixels within the first frame corresponding to the first spatiotemporal region, and wherein the change does not occur for pixels of the first frame outside the first spatiotemporal region.

16. A computer-implemented method for unsupervised training of neural network system, the computer-implemented method comprising:
receiving training data, the training data comprising a first modality and a second modality, wherein the first modality comprises image data having a plurality of frames and the second modality comprises one of audio data or text data corresponding to the image data;
extracting a set of spatiotemporal features from different spatial and temporal portions of the image data of the first modality;
extracting a set of word features from the second modality;
determining a spatiotemporal feature representation and a word feature representation at least by passing the set of spatiotemporal features and the set of word features through a plurality of cross-modal attention layers within the neural network system;
computing a contrastive loss value from the spatiotemporal feature representation and the word feature representation; and
adjusting weights within the neural network system based on the contrastive loss value.

17. The computer-implemented method of claim 16 , wherein the plurality of cross-modal attention layers alternate with at least one self-attention layer within the neural network system.

18. The computer-implemented method of claim 16 , a full attention layer is absent from the neural network system.

19. The computer-implemented method of claim 16 , wherein the word feature representation comprises a sentence-level representation corresponding to a plurality of wo

Assignees

• Adobe Inc

Inventors

• Jin Hailin
• Russell Bryan
• Tan Reuben Xin Hong

Classifications

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/22 Primary

  Matching criteria, e.g. proximity measures · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G10L15/02

  Feature extraction for speech recognition; Selection of recognition unit · CPC title