Deep navigation via a multimodal vector model

Patent metadata

Field	Value
Publication number	US-12130809-B2
Application number	US-202217947971-A
Country	US
Kind code	B2
Filing date	Sep 19, 2022
Priority date	Sep 19, 2022
Publication date	Oct 29, 2024
Grant date	Oct 29, 2024

Abstract

A multimodal embedding modifier generates a modified seed search selection embedding for providing a set of search results. The multimodal embedding modifier enhances the ability and accuracy of identifying a user's true intent when searching the online marketplace. For example, embodiments disclosed herein can allow a user to navigate multiple modalities for an item. In some embodiments, a user may select a search result corresponding to an initial search query, and further modify the selected search result by inputting a modifier (e.g., a textual modifier). The multimodal embedding modifier can be trained using a training dataset including a text embedding, an image embedding, another type of embedding, or a combination thereof.

First claim

What is claimed is:

1. A computer-implemented method comprising:
providing a first set of search results based on a search query;
receiving a seed search selection selected from the first set of search results provided;
receiving a modifier;
determining a modifier embedding for the modifier, wherein the modifier embedding is generated for the seed search selection via a multimodal model trained to generate image and text embeddings, the multimodal model comprising bidirectional encoder representations from transformers (BERT);
generating a modified seed search selection embedding by modifying a seed search selection embedding of the seed search selection using the modifier embedding; and
providing a second set of search results based on the modified seed search selection embedding.

2. The computer-implemented method of claim 1 , wherein the modifier is a natural language input.

3. The computer-implemented method of claim 1 , wherein the seed search selection comprises a plurality of search results from the first set of search results, and wherein the seed search selection embedding is determined from search result embeddings of the plurality of search results.

4. The computer-implemented method of claim 1 , wherein the seed search selection embedding comprises a text embedding from a textual element of the seed search selection and an image embedding from an image of the seed search selection.

5. The computer-implemented method of claim 4 , wherein the modified seed search selection embedding is generated by separately modifying the text embedding and the image embedding using the modifier embedding.

6. The computer-implemented method of claim 1 , wherein the seed search selection embedding is generated from a title of the seed search selection.

7. The computer-implemented method of claim 1 , wherein the modifier comprises a negative modifier, and wherein modifying the seed search selection embedding comprises removing the modifier embedding from the seed search selection embedding.

8. The computer-implemented method of claim 1 , wherein the modifier comprises a positive modifier, and wherein modifying the seed search selection embedding comprises combining the seed search selection embedding and the modifier embedding.

9. A computer system comprising:
a processor; and
a computer storage medium storing computer-useable instructions that, when used by the processor, causes the computer system to perform operations comprising:
receive a first set of search results based on a search query;
select a seed from the first set of search results for the generation of a seed embedding;
provide a modifier; and
receive a second set of search results based on a modified seed embedding of the selected seed, wherein the seed embedding of the selected seed is modified via a modifier embedding corresponding to the modifier provided, wherein the modifier embedding is generated for the seed selected via a multimodal model trained to generate image and text embeddings, the multimodal model comprising bidirectional encoder representations from transformers (BERT).

10. The computer system of claim 9 , wherein the modifier includes a positive modifier and a negative modifier, and wherein modifying the seed embedding includes:
combine at least a portion of the seed embedding and a positive modifier embedding corresponding to the positive modifier; and
remove a negative modifier embedding corresponding to the negative modifier from the seed embedding.

11. The computer system of claim 9 , wherein selecting the seed includes selecting an image of the first set of search results received.

12. The computer system of claim 9 , wherein the multimodal model is trained, at least in part, using a plurality of titles from a plurality of item listings.

13. The computer system of claim 9 , further comprising select a plurality of search results from the first set of search results as the seed, wherein the seed embedding is determined from the plurality of search results selected from the first set of search results.

14. One or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising:
receiving a seed search selection selected from a first set of search results provided via a user interface, the first set of search results based on a search query;
receiving a modifier for modifying search results for the search query based on the seed search selection;
generating, via a multimodal model trained to generate image and text embeddings, a modifier embedding for the modifier, wherein the multimodal model comprises bidirectional encoder representations from transformers;
generating a modified seed search selection embedding for the seed search selection by applying the modifier embedding to an embedding of the seed search selection; and
providing a second set of search results based on the modified seed search selection embedding.

15. The one or more computer storage media of claim 14 , wherein the second set of search results includes a plurality of search results from the first set of search results and a plurality of search results not included in the first set of search results.

16. The one or more computer storage media of claim 14 , wherein the embedding of the seed search selection includes an image embedding from an image element of the seed search selection.

17. The one or more computer storage media of claim 16 , wherein the modified seed search selection embedding is generated by modifying the image embedding using the modifier embedding, the modifier embedding corresponding to a textual modifier.

18. The one or more computer storage media of claim 14 , wherein the second set of search results are provided based on distances between the modified seed search selection embedding and the image and text embeddings generated by the multimodal model, the image and text embeddings corresponding to item listings.

19. The one or more computer storage media of claim 14 , the operations further comprising generating the modifier embedding based on applying the multimodal model to a plurality of titles from a plurality of item listings to generate the text embeddings.

20. The one or more computer storage media of claim 14 , wherein the modifier includes an image-text aggregate.

Assignees

• Ebay Inc

Inventors

• Miller Christopher
• Tandon Shubhangi
• Gopal Senthilkumar
• Kopru Selcuk

Classifications

• G06F16/248

  Presentation of query results · CPC title

• G06F16/24534 Primary

  Query rewriting; Transformation · CPC title

• G06F16/3326 Primary

  using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages · CPC title