Retrieving images that correspond to a target subject matter within a target context

Patent metadata

Field	Value
Publication number	US-11907338-B2
Application number	US-202117158170-A
Country	US
Kind code	B2
Filing date	Jan 26, 2021
Priority date	Jan 26, 2021
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

Techniques are provided herein for retrieving images that correspond to a target subject matter within a target context. Although useful in a number of applications, the techniques provided herein are particularly useful in contextual product association and visualization. A method is provided to apply product images to a neural network. The neural network is configured to classify the products in the images. The images are associated with a context representing the combination of classified products in the images. These techniques leverage both seller-provided images of products and user-generated content, which potentially includes hundreds or thousands of images of the same or similar products as the seller-provided images. A graphical user interface is configured to permit a user to select the context of interest in which to visualize the products.

First claim

What is claimed is:

1. A method for determining a context of a main product and at least one context product in an image, the method comprising:
classifying, by a product identification module, a first context product in a feature map using a first neural network having one or more product classification layers, the feature map representing features in a first image containing the main product and in a second image containing the main product;
classifying, by the product identification module, a second context product in the feature map using the first neural network;
classifying, by a prominent region detection module, a common prominent feature in the feature map using a second neural network having one or more prominent feature classification layers, the common prominent feature being different from the first context product and the second context product; and
associating, by a product association module and responsive to classifying the common prominent feature, the first image with the first context product and further associating the second image with the second context product.

2. The method of claim 1 , further comprising causing, by a product display module, the first image to be displayed on a display device in response to a user selection of the first context product, and the second image to be displayed on the display device in response to a user selection of the second context product.

3. The method of claim 1 , further comprising:
obtaining, by the product identification module and from the first neural network, a bounding box surrounding the first context product in the first image;
generating, by the product identification module, a mask corresponding to the first context product within the bounding box;
assigning, by the prominent region detection module, a weight to the mask; and
identifying, by the prominent region detection module and from the second neural network, the common prominent feature based on the mask and the weight.

4. The method of claim 3 , wherein the weight is a first weight, wherein the method further comprises assigning, by the prominent region detection module, a second weight to a region of the first image containing the main product, and wherein identifying the common prominent feature is further based on a combination of the first and second weights.

5. The method of claim 1 , further comprising obtaining, by the product identification module and from the first neural network, a first label associated with the main product, a second label associated with the first context product, and a third label associated with the second context product, wherein the first image is associated with the first context product based on the first and second labels, and wherein the second image is associated with the second context product based on the first and third labels.

6. The method of claim 5 , wherein the first neural network is a convolutional neural network (CNN) trained on a product image training dataset, the product image training dataset including the first label, the second label, and the third label.

7. The method of claim 1 , wherein the first context product is the same as the second context product, and wherein the method further comprises:
applying, by the product display module, the first image and the second image to a third neural network configured to generate an image quality score for each of the first image and the second image; and
generating, by the product display module, a thumbnail image from one of the first image and the second image having the highest image quality score.

8. The method of claim 7 , further comprising causing, by the product display module, the thumbnail image to be displayed on the display device in response to a user selection of the main product.

9. The method of claim 1 , wherein there are a plurality of first images containing the main product, and wherein the method further comprises:
associating, by the product association module and responsive to identifying the common prominent feature, each of the plurality of first images with the first context product; and
causing, by the product display module, each of the plurality of first images to be displayed on the display device in response to a user selection of the first context product.

10. A system for determining a context of a main product and at least one context product in an image, the system comprising:
a product identification module, executable by at least one processor, and configured to
classify, using a first neural network having one or more product classification layers, a first context product in a feature map, the feature map representing features in a first image containing the main product and in a second image containing the main product, and
classify, using the first neural network, a second context product in the feature map;

a prominent region detection module, executable by the at least one processor, and configured to classify, using a second neural network, a common prominent feature in the feature map, the common prominent feature being different from the first context product and the second context product;
a product association module, executable by the at least one processor responsive to classifying the common prominent feature, and configured to
associate the first image with the first context product, and
associate the second image with the second context product; and

a product display module, executable by the at least one processor, and configured to receive a user selection of the first context product or the second context product, and
cause the first image to be displayed via a display device in response to the user selection of the first context product, and the second image to be displayed via the display device in response to the user selection of the second context product.

11. The system of claim 10 , wherein the product identification module is further configured to:
obtain, from the first neural network, a bounding box surrounding the first context product in the first image;
wherein the product identification module is further configured to generate a mask corresponding to the first context product within the bounding box; and
wherein the prominent region detection module is further configured to
assign a weight to the mask, and
identify, using the second neural network, the common prominent feature based on the mask and the weight.

12. The system of claim 10 , wherein the product identification module is further configured to obtain, from the first neural network, a first label associated with the main product, a second label associated with the first context product, and a third label associated with the second context product, wherein the first image is associated with the first context product based on the first and second labels, and wherein the second image is associated with the second context product based on the first and third labels.

13. The system of claim 10 , wherein the first neural network is a convolutional neural network (CNN) trained on a product image training dataset, the product image training dataset including the first label, the second label, and the third label.

14. The system of claim 10 , wherein the first context product is the same as the second context product, and wherein the product display module is further configured to:
apply the first image and the second image to a third neural network configured to generate an image quality score for each of the first image and the second image;
generate a thumbnail image from one of the first image and the second image having the highest image quality score;

Assignees

• Adobe Inc

Inventors

• Singhal Gourav
• Gupta Sourabh
• Sharma Mrinal Kumar

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06F18/24 Primary

  Classification techniques · CPC title

• G06F3/14 Primary

  Digital output to display device {; Cooperation and interconnection of the display device with other functional units} · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title