Correlating image annotations with foreground features

Patent metadata

Field	Value
Publication number	US-11657084-B2
Application number	US-202017107483-A
Country	US
Kind code	B2
Filing date	Nov 30, 2020
Priority date	Sep 5, 2013
Publication date	May 23, 2023
Grant date	May 23, 2023

Abstract

A machine may be configured to execute a machine-learning process for identifying and understanding fine properties of various items of various types by using images and associated corresponding annotations, such as titles, captions, tags, keywords, or other textual information applied to these images. By use of a machine-learning process, the machine may perform property identification accurately and without human intervention. These item properties may be used as annotations for other images that have similar features. Accordingly, the machine may answer user-submitted questions, such as “What do rustic items look like?,” and items or images depicting items that are deemed to be rustic can be readily identified, classified, ranked, or any suitable combination thereof.

First claim

What is claimed is:

1. A method, comprising:
receiving a query, the query comprising a query image that depicts an item, the query image comprising a query background and a query foreground;
segmenting, by one or more hardware processors, the query image into at least a first portion representing the query background and a second portion representing the query foreground;
calculating a query feature descriptor based on the query foreground;
determining, by the one or more hardware processors, that the query feature descriptor matches a reference feature descriptor of a generated data structure, the generated data structure comprising a plurality of reference feature descriptors and a plurality of correlated item annotations associated with one or more reference images depicting one or more additional items, wherein the reference feature descriptor corresponds to a reference image of the one or more reference images;
obtaining, based on the query feature descriptor matching the reference feature descriptor, an item annotation of the plurality of correlated item annotations, the item annotation correlated with the reference feature descriptor, wherein the item annotation corresponds to the reference image of the one or more reference images; and
providing a response to the query based on the item annotation, the response to the query comprising a suggestion that the item annotation characterizes the query image.

2. The method of claim 1 , further comprising:
segmenting the query image into the query background and the query foreground defined by an outline of the item; and
partitioning the query foreground into multiple sections, wherein calculating the query feature descriptor comprises calculating a local feature descriptor based on only one section among the multiple sections partitioned from the query foreground.

3. The method of claim 1 , further comprising:
providing a user with the response to the query based on the generated data structure, wherein the received query includes a query annotation;
determining that the query annotation matches the item annotation correlated with the reference feature descriptor of the generated data structure; and
obtaining the reference feature descriptor from the generated data structure based on the query annotation matching the item annotation.

4. The method of claim 3 , wherein: the response to the query further comprises a suggestion that the query image is characterized by the query annotation.

5. The method of claim 3 , wherein the response to the query further comprises the plurality of correlated item annotations.

6. The method of claim 1 , wherein: the reference feature descriptor of the generated data structure is a color descriptor calculated from a color of a reference item whose outline defines a segmented foreground.

7. The method of claim 1 , wherein: the reference feature descriptor of the generated data structure is a shape descriptor calculated from an outline of a reference item whose outline defines a segmented foreground.

8. The method of claim 1 , wherein: the item annotation is an n-gram included within a caption of the query image, the caption being submitted by a seller of the item depicted in the query image.

9. The method of claim 1 , wherein: the item annotation is a keyword submitted as a tag for the query image by a seller of the item.

10. The method of claim 1 , wherein: the item annotation is a name-value pair that specifies an attribute of the item depicted in the query image and whose outline defines the query foreground.

11. A system comprising:
one or more hardware processors configured to perform operations comprising:
receiving a query, the query comprising a query image that depicts an item, the query image comprising a query background and a query foreground;
segmenting the query image into at least a first portion representing the query background and a second portion representing the query foreground;
calculating a query feature descriptor based on the query foreground;
determining that the query feature descriptor matches a reference feature descriptor of a generated data structure, the generated data structure comprising a plurality of reference feature descriptors and a plurality of correlated item annotations associated with one or more reference images depicting one or more additional items, wherein the reference feature descriptor corresponds to a reference image of the one or more reference images;
obtaining, based on the query feature descriptor matching the reference feature descriptor, an item annotation of the plurality of correlated item annotations, the item annotation correlated with the reference feature descriptor, wherein the item annotation corresponds to the reference image of the one or more reference images; and
providing a response to the query based on the item annotation, the response to the query comprising a suggestion that the item annotation characterizes the query image.

12. The system of claim 11 , wherein the operations further comprise:
segmenting the query image into the query background and the query foreground defined by an outline of the item; and
partitioning the query foreground into multiple sections, wherein calculating the query feature descriptor comprises calculating a local feature descriptor based on only one section among the multiple sections partitioned from the query foreground.

13. The system of claim 11 , wherein the operations further comprise:
providing a user with the response to the query based on the generated data structure, wherein the received query includes a query annotation;
determining that the query annotation matches the item annotation correlated with the reference feature descriptor of the generated data structure; and
obtaining the reference feature descriptor from the generated data structure based on the query annotation matching the item annotation.

14. The system of claim 13 , wherein: the response to the query further comprises a suggestion that the query image is characterized by the query annotation.

15. The system of claim 13 , wherein the response to the query further comprises the plurality of correlated item annotations.

16. The system of claim 11 , wherein: the reference feature descriptor of the generated data structure is a color descriptor calculated from a color of a reference item whose outline defines a segmented foreground.

17. The system of claim 11 , wherein: the reference feature descriptor of the generated data structure is a shape descriptor calculated from an outline of a reference item whose outline defines a segmented foreground.

18. The system of claim 11 , wherein: the item annotation is an n-gram included within a caption of the query image, the caption being submitted by a seller of the item depicted in the query image.

19. The system of claim 11 , wherein: the item annotation is a keyword submitted as a tag for the query image by a seller of the item.

20. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving a query, the query comprising a query image that depicts an item, the query image comprising a query background and a query foreground;
segmenting the query image into at least a first portion representing the query background and a second portion representing the query foreground;
calculating a query feature descriptor based on the query foreground;
determining that

Assignees

• Ebay Inc

Inventors

• Bhardwaj Anurag
• Piramuthu Robinson
• Ordonez-Roman Vicente
• Jagadeesh Vignesh
• Di Wei

Classifications

• G06F16/58 Primary

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title

• G06F16/5838 Primary

  using colour · CPC title

• G06F16/5846

  using extracted text · CPC title

• G06F16/532

  Query formulation, e.g. graphical querying · CPC title

• G06F16/5862

  using texture · CPC title