Generating search strings and refinements from an image

Patent metadata

Field	Value
Publication number	US-9875258-B1
Application number	US-201514973578-A
Country	US
Kind code	B1
Filing date	Dec 17, 2015
Priority date	Dec 17, 2015
Publication date	Jan 23, 2018
Grant date	Jan 23, 2018

Abstract

Approaches include using a machine learning-based approach to generating search strings and refinements based on a specific item represented in an image. For example, a classifier that is trained on descriptions of images can be provided. An image that includes a representation of an item of interest is obtained. The image is analyzed using the classifier algorithm to determine a first term representing a visual characteristic of the image. Then, the image is analyzed again to determine a second term representing another visual characteristic of the image based at least in part on the first term. Additional terms can be determined to generate a description of the image, including characteristics of the item of interest. Based on the determined characteristics of the item of interest, a search query and one or more refinements can be generated.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
obtaining an image by a camera of a computing device, the image including a representation of an object;
analyzing the image to determine a product category associated with the object;
analyzing the image using a classifier algorithm to determine a first term representing a first visual characteristic of the image;
analyzing the image using the classifier algorithm with the first term to determine a second term representing a second visual characteristic of the image, the second term including one or more words;
determining that the one or more words of the second term is a sequence of words describing visual characteristics associated with the object, the sequence of words satisfying a search condition;
generating, in response to the sequence of words satisfying the search condition, a search string query, one or more refinements, and one or more key words based at least in part on the product category and the sequence of words;
determining a set of search results corresponding to the search string query; and
displaying the set of search results with the one or more refinements and the one or more key words for the product category on the computing device, the one or more refinements and the one or more key words configured to be selectable by a user of the computing device, the search string query being configured to be editable by the user in response to a selection of one of the one or more refinements and the one or more key words.

2. The computer-implemented method of claim 1 , further comprising:
analyzing the image using a recurrent neural network; and
training the recurrent neural network using descriptions of a set of images to predict strings of terms associated with images.

3. The computer-implemented method of claim 1 , further comprising:
training a neural network to predict feature vectors associated with images, the feature vectors describing categories of objects;
analyzing the image using the neural network to generate a feature vector that represents the product category associated with the object; and
analyzing the image using the classifier algorithm to determine the second term based at least in part on the first term and the feature vector.

4. The computer-implemented method of claim 1 , wherein the set of search results is a first set of search results, and wherein the method further comprises:
generating a second search string query, a second set of one or more refinements, and a second set of one or more key words based on characteristics of the object;
determining a second set of search results corresponding to the second search string query, the product category, the second set of one or more refinements, and the second set of one or more key words;
determining that the second set of search results is more closely associated with the object than the first set of search results; and
displaying the second set of search results to the computing device.

5. A computer-implemented method, comprising:
analyzing an image including a representation of an object using a first classifier to determine a product category associated with the object;
analyzing the image using a second classifier algorithm to determine a term representing a visual characteristic of the image;
analyzing the image using the second classifier algorithm with the term to determine a sequence of words describing visual characteristics associated with the object;
determining that the sequence of words satisfies a search condition;
generating, in response to the sequence of words satisfying the search condition, a search string query that includes a subset of the sequence of words and search string refinement terms associated with the product category and the sequence of words;
determining a set of search results based at least in part on the search string query that includes an item from a catalog of items; and
displaying the set of search results and the search string refinement terms on a computing device, the search string refinement terms being selectable, the search string query being configured to be editable in response to a selection of one of the search string refinement terms.

6. The computer-implemented method of claim 5 , further comprising:
causing the search string query to be transmitted to obtain the set of search results.

7. The computer-implemented method of claim 5 , further comprising:
analyzing the image using the second classifier algorithm to determine the sequence of words further based at least in part on the product category.

8. The computer-implemented method of claim 5 , wherein the set of search results is a first set of search results, and wherein the method further comprises:
generating a second search string query, an additional set of search string refinement terms based on the visual characteristics of the object;
determining a second set of search results corresponding to the second search string query; and
displaying the second set of search results and the additional set of search string refinement terms to the computing device.

9. The computer-implemented method of claim 5 , further comprising:
displaying a subset of the set of search results on the computing device in response to one or more of: a selection of a search string refinement term and an edit of the search string query by a user.

10. The computer-implemented method of claim 5 , further comprising:
obtaining a plurality of images;
analyzing the plurality of images to determine a description associated with respective images of the plurality of images; and
training a recurrent neural network using the plurality of images and associated descriptions to determine words representing visual characteristics in the plurality of images.

11. The computer-implemented method of claim 5 , further comprising:
training a neural network to predict feature vectors associated with images, a feature vector describing a categories of an object represented in an image;
analyzing the image using the neural network to generate a feature vector that represents the product category associated with the object; and
analyzing the image using the second classifier algorithm to determine the sequence of words based at least in part on the term and the feature vector.

12. A system, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the system to:
analyze an image including a representation of an object using a first classifier to determine a product category associated with the object;
analyze the image using a second classifier algorithm to determine a term representing a visual characteristic of the image;
analyzing the image using the second classifier algorithm with the term to determine a sequence of words describing visual characteristics associated with the object;
determine that the sequence of words satisfies a search condition;
generate, in response to the sequence of words satisfying the search condition, a search string query that includes a subset of the sequence of words and search string refinement terms associated with the product category and the sequence of words;
determine a set of search results based at least in part on the search string query that includes an item from a catalog of items; and
display the set of search results and the search string refinement terms on a computing device, the search string refinement terms being selectable, the search string query being configured to be editable in response to a selection of one of the search string refinement terms.

13. Th

Assignees

• A9 Com Inc

Inventors

• Hsiao Edward
• Gray Douglas Ryan

Classifications

• G06N3/08

  Learning methods · CPC title

• G06F16/532 Primary

  Query formulation, e.g. graphical querying · CPC title

• G06F16/90332

  Natural language query formulation or dialogue systems · CPC title

• G06F16/5846

  using extracted text · CPC title

• G06F16/248

  Presentation of query results · CPC title