Language translation based on search results and user interaction data

Patent metadata

Field	Value
Publication number	US-10776707-B2
Application number	US-201615064523-A
Country	US
Kind code	B2
Filing date	Mar 8, 2016
Priority date	Mar 8, 2016
Publication date	Sep 15, 2020
Grant date	Sep 15, 2020

Abstract

Various aspects of the subject technology relate to systems, methods, and machine-readable media for language translation based on image search similarities. These aspects include an image retrieval system using a convolutional neural network that is trained to identify a correlation between an image and a language term, and using an image search engine to search against images corresponding to visual words that are responsive to a given search query in a given spoken language. These aspects include access to interaction probability data that identifies user interaction probabilities for the visual words to determine a correlation between the input language terms of the search query and the rate at which users interact with images of a corresponding visual word that is responsive to the search query. The system then provides a prioritized listing of images that is responsive to the given search query based on the identified user interaction probabilities.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving a user input identifying a search query associated with a language;
identifying one or more visual words corresponding to the search query based on a historical search query in the language, wherein the one or more visual words corresponds to one or more semantic concepts of the search query;
obtaining images associated with the one or more visual words from a collection of images;
providing a listing of the images that are prioritized, prior to receiving the user input, based on a user interaction probability of the one or more visual words from the historical search query, the user interaction probability indicative of an interaction distribution of images corresponding to the one or more visual words, the interaction distribution of images identifying respective rates at which users of the language interact with the images corresponding to each of the one or more visual words, wherein the listing of the images comprises a first quantity of images based on the user interaction probability associated with a first visual word of the one or more visual words and a second quantity of images based on the user interaction probability associated with a second visual word of the one or more visual words, wherein the first quantity of images and the second quantity of images are provided in proportion to the user interaction probability;
detecting subsequent user interactions with the images associated with the one or more visual words; and
updating the user interaction probability of the images corresponding to each of the one or more visual words with the subsequent user interactions that are detected, wherein the search query comprises a language term and identifying the one or more visual words comprises selecting a centroid in a cluster of vectors, each vector associated with an image from the collection of images that includes features indicative of the language term.

2. The computer-implemented method of claim 1 , wherein the identifying the one or more visual words comprises:
obtaining, in response to the user input, the historical search query in the language;
parsing the historical search query to identify mapping information, the mapping information indicating an association between the one or more visual words and the search query; and
identifying the user interaction probability associated with the one or more visual words from the mapping information.

3. The computer-implemented method of claim 1 , wherein the interaction distribution of images identifies respective download rates for users of the language corresponding to each of the one or more visual words.

4. The computer-implemented method of claim 1 , further comprising:
obtaining the historical search queries in a target language, the historical search queries including interaction data that identifies the subsequent user interactions with one or more image search results responsive to at least one of the historical search queries;
identifying the one or more visual words in the historical search queries of the target language;
determining one or more language terms in the target language that correspond to the one or more visual words based on the historical search queries and the subsequent user interactions; and
providing a listing of the one or more language terms in the target language in response to the search query.

5. The computer-implemented method of claim 4 , further comprising:
providing a plurality of translation suggestions identifying a translation from the language to the target language, the plurality of translation suggestions indicating language words in the target language that are respectively associated with one of the one or more visual words.

6. The computer-implemented method of claim 5 , wherein the plurality of translation suggestions include a listing of the language words that are prioritized according to how closely each of the language words corresponds to the one of the one or more visual words.

7. The computer-implemented method of claim 1 , further comprising:
processing search data of users in a first language;
identifying a first image corresponding to the first visual word being download in response to the search query at a first download rate;
identifying a second image corresponding to the second visual word being download in response to the search query at a second download rate; and
mapping one or more search terms of the search query to the first visual word and the second visual word into mapping information associated with the first language, including indications of the first download rate and the second download rate respectively with the first visual word and the second visual word, wherein a mapping occurs prior to the receiving the user input.

8. The computer-implemented method of claim 7 , further comprising:
detecting interactions with the first image and the second image, subsequent to a processing of a search data;
determining that the first image and the second image are download at a rate different than the first download rate and the second download rate; and
modifying the mapping information to include modified download rates for the first image and the second image based on the interactions.

9. The computer-implemented method of claim 1 , further comprising:
providing a set of training images to a convolutional neural network;
providing semantic data identifying the one or more semantic concepts to the convolutional neural network;
providing mapping data identifying relationships between the set of training images and the one or more semantic concepts, wherein the convolutional neural network processes the set of training images and the mapping data to learn to identify features relating to at least one of the one or more semantic concepts;
submitting a plurality of images from a collection of images to the convolutional neural network that is configured to analyze image pixel data for each of the plurality of images to identify features that relate to at least one of the one or more semantic concepts;
generating multiple vectors for each of the plurality of images using the features; and
forming a cluster with the vectors to find the visual word.

10. The computer-implemented method of claim 9 , further comprising:
tagging each of the plurality of images with metadata identifying one or more keywords in one or more spoken languages.

11. A system comprising:
one or more processors;
a computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium including instructions that, when executed by the one or more processors, cause the one or more processors to:
receive a search query identifying one or more search terms in a language selected from multiple languages, for initiating an image search associated with a target language, the target language being different than the language;
identify, prior to receiving the search query, a user interaction probability for a plurality of images from a collection of images that are responsive to historical search queries in the language, the user interaction probability indicative of an interaction distribution of images corresponding to a one or more visual words, the interaction distribution of images identifying respective rates at which users of the language interact with the images corresponding to each of the one or more visual words, wherein the one or more visual words corresponds to one or more semantic concepts of the search query;
identify a subset of the plurality of images corresponding to the one or more visual words that are responsive to a historical search query iden

Assignees

• Shutterstock Inc

Inventors

• Chester David
• Hurst Nathan

Classifications

• G06N7/01 Primary

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title