Complementary item recommendations based on multi-modal embeddings

Patent metadata

Field	Value
Publication number	US-12282946-B2
Application number	US-202017011543-A
Country	US
Kind code	B2
Filing date	Sep 3, 2020
Priority date	Sep 5, 2019
Publication date	Apr 22, 2025
Grant date	Apr 22, 2025

Abstract

Systems and methods for providing suggestions of complementary products responsive to an anchor product are disclosed. The method includes receiving a selection of an anchor product. A similarity score between text embeddings of the anchor product and text embeddings of a plurality of products in a product database is calculated. A similarity score between an image feature of the anchor product and an image feature of the plurality of products in the product database is calculated. A weighted score between the two similarity scores as calculated for the anchor product and the plurality of products in the product database is calculated. At least one of the products from the product database having a highest weighted score is selected and returned responsive to the selection of the anchor product.

First claim

What is claimed is:

1. A system, comprising:
a server; and
a product recommendation system,
wherein the product recommendation system is in electronic communication with the server,
wherein the product recommendation system comprises:
a processor; and
a memory storing instructions that, when executed by the processor, cause the system to perform a method, comprising:
receiving, by the product recommendation system from a user device, a selection of an anchor product;
training, by the processor, a machine learning model with product information for a plurality of products in a product database;
generating, from the machine learning model, text embeddings comprising vector descriptions of text associated with each product of the plurality of products in the product database for each product of the plurality of products, wherein the machine learning model comprises a multi-stream network;
calculating, by the product recommendation system, a similarity score between text embeddings of the anchor product and text embeddings of a plurality of products in a product database;
calculating, by the product recommendation system, a similarity score between an image feature of the anchor product and an image feature of the plurality of products in the product database,
 wherein the image features comprise vector descriptions of color content of the images;
calculating, by the product recommendation system, a weighted score between the two similarity scores as calculated for the anchor product and the plurality of products in the product database, wherein a weight of the similarity score between the image feature of the plurality of products in the product database and a weight of the similarity score between text embeddings of the anchor product and text embeddings of the plurality of products in the product database is varied using a product type determined from the text embeddings of the anchor product;
selecting, by the product recommendation system, at least one of the products from the product database having a highest weighted score; and
returning, to the user device by the product recommendation system, the at least one of the products as selected responsive to the selection of the anchor product, automatically causing the user device to display on a display of the user device the at least one of the products as selected responsive to the selection of the anchor product.

2. The system of claim 1 , wherein the weighted score includes a text weight value based on a product type and an image weight value based on the product type.

3. The system of claim 2 , comprising a first product type and a second product type, the first product type having a greater importance of visual features and a lower importance of textual features, the second product type having a lower importance of visual features and a greater importance of textual features.

4. The system of claim 3 , wherein the text weight value of the first product type is lower than the text weight value of the second product type, and the image weight value of the first product type is greater than the image weight value of the second product type.

5. The system of claim 1 , wherein the image feature includes a red-green-blue (RGB) color histogram on the image.

6. The system of claim 5 , comprising determining, by the product recommendation system, the RGB color histogram for a foreground of the anchor product and the plurality of products in the product database.

7. The system of claim 5 , wherein the RGB channels of the image include 8 bins per channel to obtain a 512-dimensional feature vector for the anchor product and the plurality of products in the product database.

8. The system of claim 1 , wherein the selecting at least one of the products from the product database having a highest weighted score includes selecting a plurality of products and returning at least one of the plurality of products having a different product type than a product type of the anchor product.

9. A method, comprising:
receiving, by a product recommendation system from a user device, a selection of an anchor product;
training, by the product recommendation system, a machine learning model with product information for a plurality of products in a product database;
generating, from the machine learning model, text embeddings comprising vector descriptions of text associated with each product of the plurality of products in the product database for each product of the plurality of products, wherein the machine learning model comprises a multi-stream network;
calculating, by the product recommendation system, a similarity score between text embeddings of the anchor product and text embeddings of a plurality of products in a product database,
wherein the text embeddings comprise vector descriptions of text associated with each product of the plurality of products in the product database,
wherein the text embeddings for each product of the plurality of products are generated from a machine learning model, wherein the machine learning model is trained based on product information for each of the plurality of products in the product database for each of the plurality of products;
calculating, by the product recommendation system, a similarity score between an image feature of the anchor product and an image feature of the plurality of products in the product database;
calculating, by the product recommendation system, a weighted score between the two similarity scores as calculated for the anchor product and the plurality of products in the product database,
wherein a weight of the similarity score between the image feature of the plurality of products in the product database and a weight of the similarity score between text embeddings of the anchor product and text embeddings of the plurality of products in the product database is varied using a product type determined from the text embeddings of the anchor product;
selecting, by the product recommendation system, at least one of the products from the product database having a highest weighted score; and
returning, to the user device by the product recommendation system, the at least one of the products as selected responsive to the selection of the anchor product, automatically causing the user device to display on a display of the user device the at least one of the products as selected responsive to the selection of the anchor product.

10. The method of claim 9 , wherein calculating the similarity score between text embeddings of the anchor product and text embeddings of the plurality of products in a database includes calculating a cosine similarity score between the text embeddings of the anchor product and the text embeddings of the plurality of products in the product database.

11. The method of claim 9 , wherein calculating the similarity score between the image feature of the anchor product and the image feature of the plurality of products in the product database includes calculating a cosine similarity score between the image features of the anchor product and the image features of the plurality of products in the product database.

12. The method of claim 9 , comprising separating the background and foreground of the image.

13. The method of claim 12 , wherein the separating comprises a mean adaptive threshold.

14. The method of claim 12 , wherein the image feature includes a red-green-blue (RGB) color histogram on the image, the method comprising determining the RGB color histogram for the foreground of the image following the separating the background and the foreground of the image.

15. The method of claim 1

Assignees

• Home Depot Product Authority Llc

Inventors

• Al Jadda Khalifeh
• Ahsan Unaiza
• Wu San He
• Qu Huiming

Classifications

• G06Q30/0643

  graphically representing goods, e.g. 3D product representation · CPC title

• G06F16/906

  Clustering; Classification · CPC title

• G06F16/9038

  Presentation of query results · CPC title

• G06Q30/0629

  by pre-processing results, e.g. ranking or ordering results · CPC title

• G06Q30/0631 Primary

  Recommending goods or services · CPC title