Automatic clustering and mapping of user generated content with curated content

Patent metadata

Field	Value
Publication number	US-11748796-B2
Application number	US-202016824180-A
Country	US
Kind code	B2
Filing date	Mar 19, 2020
Priority date	Mar 19, 2020
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

Methods, systems, and non-transitory computer readable media are disclosed for determining a sub-set of user-submitted images that are similar to a curated image and presenting the sub-set of user-submitted images in connection with the curated image. The disclosed system presents a curated image depicting a product via a graphical user interface (e.g., on an e-commerce platform). In one or more embodiments, the disclosed system extracts feature vectors from the curated image and a plurality of user-submitted images. The disclosed system compares the feature vectors from the curated image and the plurality of user-submitted images to determine a sub-set of user-submitted images that are similar to the curated image. The disclosed system presents the sub-set of user-submitted images based on a user selection of the curated image.

First claim

What is claimed is:

1. A non-transitory computer readable medium for presenting clustered images, the non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:
present, via a graphical user interface at a user client device, curated images displaying a product;
extract, utilizing a machine learning algorithm, one or more frequency domain descriptors from the curated images;
compile, utilizing the machine learning algorithm, the one or more frequency domain descriptors into feature vectors from the curated images;
extract, utilizing the machine learning algorithm, one or more frequency domain descriptors from a plurality of user-submitted images displaying the product;
compile, utilizing the machine learning algorithm, the one or more frequency domain descriptors from the plurality of user-submitted images displaying the product into feature vectors from the plurality of user-submitted images displaying the product;
determine a sub-set of the plurality of user-submitted images for each curated image that show a view of the product positioned at an angle within a threshold distance to a positioned angle of the product within a given curated image by comparing the feature vectors of the plurality of user-submitted images with the feature vectors of the curated images;
determine an additional sub-set of user-submitted images that show the view of the product positioned at one or more additional angles outside of the threshold distance of the positioned angle of the product within the curated images by comparing the feature vectors of the plurality of user-submitted images with the feature vectors of the curated images;
receive, via the graphical user interface, a user selection of a curated image;
present, via the graphical user interface and based on the user selection of the curated image, the sub-set of the user-submitted images that show the view of the product positioned at the angle within the threshold distance to the positioned angle of the product within the curated image;
receive, via the graphical user interface, a user selection of an additional views element; and
present, via the graphical user interface and based on the user selection of the additional views element, one or more of the user-submitted images in the additional sub-set of user-submitted images that show the view of the product positioned at the one or more additional angles outside of the threshold distance of the positioned angle of the product within the curated images.

2. The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to extract the one or more frequency domain descriptors from the curated images by generating object descriptors for the product in the curated images.

3. The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to extract the one or more frequency domain descriptors from the plurality of user-submitted images displaying the product by:
generating object bounding boxes and labels for objects in the plurality of user-submitted images, wherein the object bounding boxes comprise product bounding boxes and product labels corresponding to the product;
cropping the product bounding boxes; and
extracting frequency domain descriptors from the product bounding boxes.

4. The non-transitory computer readable medium as recited in claim 3 , wherein generating the object bounding boxes and the labels comprises utilizing a region proposal neural network to identify the object bounding boxes and the labels.

5. The non-transitory computer readable medium as recited in claim 3 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate confidence scores corresponding to the labels.

6. The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the view of the product positioned at the angle of sub-set of the user-submitted images are within a threshold distance to the positioned angle of the product within the curated image by:
mapping a feature vector from the curated image and the feature vectors from the plurality of user-submitted images in a vector space;
determining distances between the feature vector from the curated image and each of the feature vectors from the plurality of user-submitted images in the vector space; and
determining that distances between the feature vectors of the sub-set of the user-submitted images and the feature vector from the curated image are within a threshold distance.

7. The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the view of the product positioned at one or more additional angles of the additional sub-set of user-submitted images is outside of the threshold distance of the positioned angle of the product within the curated images by:
mapping the feature vectors from the curated images and the feature vectors from the plurality of user-submitted images in a vector space;
determine distances between the feature vectors from the curated images and each of the feature vectors from the plurality of user-submitted images in the vector space;
determine that distances between the feature vectors from the curated images and the feature vectors from one or more user-submitted images of the plurality of user-submitted images exceed a threshold distance; and
generate a new cluster comprising the one or more user-submitted images.

8. The non-transitory computer readable medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to present the sub-set of the user-submitted images by:
generating aesthetic values for each user-submitted image of the sub-set of the user-submitted images;
ordering the sub-set of the user-submitted images based on the aesthetic values; and
presenting the ordered sub-set of the user-submitted images.

9. The non-transitory computer readable medium as recited in claim 1 , wherein the curated image comprises a product image created by a seller of the product.

10. A system comprising:
at least one non-transitory computer readable medium storing a curated image displaying a product and a plurality of user-submitted images displaying the product; and
at least one server configured to cause the system to:
present, via a graphical user interface at a user client device, the curated image displaying a first angled position of the product in a first view;
extract, utilizing a machine learning algorithm, one or more frequency domain descriptors from the curated image;
compile, utilizing the machine learning algorithm, the one or more frequency domain descriptors into a scale and rotation invariant feature vector from the curated image;
extract, utilizing the machine learning algorithm, one or more frequency domain descriptors from the plurality of user-submitted images displaying product;
compile, utilizing the machine learning algorithm, the one or more frequency domain descriptors into scale and rotation invariant feature vectors from the plurality of user-submitted images displaying the first angled position of the product in the first view and one or more additional angled positions of the product in one or more additional vi

Assignees

• Adobe Inc

Inventors

• Gupta Sourabh
• Sharma Mrinal Kumar
• Singhal Gourav

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06Q30/0623 Primary

  by investigating goods or services · CPC title

• G06F16/583

  using metadata automatically derived from the content · CPC title