Contextual grounding of natural language phrases in images

Patent metadata

Field	Value
Publication number	US-11620814-B2
Application number	US-202017014984-A
Country	US
Kind code	B2
Filing date	Sep 8, 2020
Priority date	Sep 12, 2019
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

Aspects of the present disclosure describe systems, methods and structures providing contextual grounding—a higher-order interaction technique to capture corresponding context between text entities and visual objects.

First claim

The invention claimed is:

1. A method for text-image retrieval including text and image branches, said method comprising:
receiving as input a text query and an image;
parsing the input text query into tokens and converting them to entity embedding vectors;
locating visual object candidates in the input image;
scoring correspondences between the entity embeddings and visual object candidates;
providing, visualized in a bounding box, the object corresponding to the query text entity with the highest probability score, to a user of the system;
pre-training the text branch utilizing a BERT, Bidirectional Encoder Representations from Transformers, base model;
receiving, by the image branch, region of interest (RoI) features as input objects from an object detector;
training, a two-layer multi-layer perceptron (MLP) to generate spatial embedding given absolute spatial information of the RoI location and size normalized to the entire image;
adding, by both branches, positional and spatial embedding to tokens and RoIs respectively as input to a first interaction layer of the MLP; and
performing, at each layer of the MLP, self-attenuation by each hidden representation to each other to generate a new hidden representation as layer output;
wherein no specific embedding or object feature extraction is used in the method.

2. The method of claim 1 further comprising:
providing, at the end of each branch, a final hidden state to a ground head to provide cross-modal attention responses with text entity hidden states as queries and image object hidden representations as keys.

3. The method of claim 2 wherein matching correspondences are determined from the attention responses.

4. The method of claim 3 further comprising
back propagating a mean binary cross entropy loss per entity if the correspondence(s0 does not match a ground truth.

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Lai Farley
• Kadav Asim
• Xie Ning

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06F40/284

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title