Systems and methods for utilizing a convolutional neural network architecture for visual product recommendations

Patent metadata

Field	Value
Publication number	US-10275820-B2
Application number	US-201715420885-A
Country	US
Kind code	B2
Filing date	Jan 31, 2017
Priority date	Jan 31, 2017
Publication date	Apr 30, 2019
Grant date	Apr 30, 2019

Abstract

Systems and methods including one or more processing modules and one or more non-transitory storage modules storing computing instructions configured to run on the one or more processing modules and perform acts of accessing an online catalog for an online retailer comprising a plurality of digital images of a plurality of items for sale by the online retailer, training a two-branch a Siamese convolutional neural network (CNN) model to determine a similarity between two digital images of the plurality of digital images, receiving one or more digital images of a new item for the online catalog, determining, using the two-branch Siamese CNN model and the one or more digital images of the new item, a similar item of the plurality of items to which the new item is most similar, and coordinating a display of the new item on a webpage based on a ranking of the similar item.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
one or more non-transitory storage devices storing computing instructions configured to run on the one or more processors and perform acts of:
accessing an online catalog for an online retailer, the online catalog comprising a plurality of digital images of a plurality of items for sale by the online retailer;
training a two-branch Siamese convolutional neural network (CNN) model, using the plurality of digital images and user session data from a plurality of users of the online catalog, to determine a similarity between two digital images of the plurality of digital images;
receiving one or more digital images of a new item for the online catalog;
determining, using the two-branch Siamese CNN model and the one or more digital images of the new item, at least one similar item of the plurality of items to which the new item is most similar; and
coordinating a display of the new item on a webpage based on a ranking of the at least one similar item, wherein:
each branch of the two-branch Siamese CNN model comprises a plurality of layers that transform each digital image of the plurality of digital images into a 100-dimensional vector; and
training the two-branch Siamese CNN model comprises:
inputting a plurality of pairs of digital images of the plurality of digital images into the two-branch Siamese CNN model, each pair of the plurality of pairs of digital images being labeled as similar or dissimilar before being input into the two-branch Siamese CNN model;
determining a distance between the 100-dimensional vector of each digital image of each pair of the plurality of pairs of digital images; and
determining a contrastive loss for each pair of the plurality of pairs of digital images using a first set of rules, the contrastive loss comprising a number indicating the similarity of a pair of digital images.

2. The system of claim 1 , wherein:
an input for the two-branch Siamese CNN model comprises the pair of digital images from the plurality of digital images; and
an output of the two-branch Siamese CNN model comprises the number indicating the similarity of the pair of digital images.

3. The system of claim 2 , wherein the plurality of layers of each branch of the two-branch Siamese CNN model comprise, in order of application to the input: a first convolutional layer, a first maximum pool layer, a first normalization layer, a second convolutional layer, a second maximum pool layer, a second normalization layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, a dropout layer, a rectified linear unit layer, and a fourth fully connected layer.

4. The system of claim 1 , wherein the first set of rules for determining the contrastive loss of each pair of the plurality of pairs of digital images comprise:
L (θ)=Σ (x p, x q ) L 2 ( x p ,x q ) 2 +Σ (x m, x n ) (margin −L 2 ( x m ,x n )) 2

where L(θ) is the contrastive loss, Σ (x p , x q) L 2 (x p , x q ) 2 is a first penalty for one or more first pairs of digital images x p and x q of the plurality of pairs of digital images labeled as similar but determined by the two-branch Siamese CNN model to be dissimilar, Σ (x m, x n) (margin−L 2 (x m , x n )) 2 is a second penalty for one or more second pairs of digital images x m and x n of the plurality of digital images labeled as dissimilar but determined by the two-branch Siamese CNN model to be similar, margin is 1, and L 2 is a Euclidian distance.

5. The system of claim 1 , wherein the one or more non-transitory storage devices storing the computing instructions are further configured to run on the one or more processors and perform acts of:
combining the similarity between the two digital images of the plurality of digital images with one or more content-based features and one or more feedback-based features to determine at least one additional similar item of the plurality of items similar to an additional item of the plurality of items; and
coordinating a display of the additional item on the webpage based on a ranking of the at least one additional similar item.

6. The system of claim 5 , wherein combining the similarity between two digital images of the plurality of digital images with the one or more content-based features and the one or more feedback-based features comprises:
indexing each 100-dimensional vector of the plurality of digital images onto an index;
determining that the 100-dimensional vector of one or more digital images of the plurality of digital images is within a predetermined distance on the index of the 100-dimensional vector of a digital image of the plurality of digital images associated with a particular item of the plurality of items;
constraining the particular item with each item of the plurality of items associated with the one or more digital images of the plurality of digital images within the predetermined distance on the index to form one or more constrained pairs;
accessing additional user session data for a predetermined period of time relating to each of the one or more constrained pairs, the additional user session data comprising the one or more content-based features and the one or more feedback-based features;
creating training data using the additional user session data for the predetermined period of time relating to each of the one or more constrained pairs; and
inserting the training data into an item ranking model configured to rank one or more items of the plurality of items related to the particular item.

7. The system of claim 1 , wherein:
an input for the two-branch Siamese CNN model comprises the pair of digital images from the plurality of digital images;
an output of the two-branch Siamese CNN model comprises the number indicating the similarity of the pair of digital images;
the plurality of layers of each branch of the two-branch Siamese CNN model comprise, in order of application to the input: a first convolutional layer, a first maximum pool layer, a first normalization layer, a second convolutional layer, a second maximum pool layer, a second normalization layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, a dropout layer, a rectified linear unit layer, and a fourth fully connected layer;
the first set of rules for determining the contrastive loss of each pair of the plurality of pairs of digital images comprise:
L (θ)=Σ (x p, x q ) L 2 ( x p ,x q ) 2 +Σ (x m, x n ) (margin −L 2 ( x m ,x n )) 2

where L(θ) is the contrastive loss, Σ (x p, x q ) L 2 (x p , x q ) 2 is a first penalty for one or more first pairs of digital images x p and x q of the plurality of pairs of digital images labeled as similar but determined by the two-branch Siamese CNN model to be dissimilar, Σ (x m, x n ) (margin−L 2 (x m , x n )) 2 is a second penalty for one or more second pairs of digital images x m and x n of the plurality of digital images labeled as dissimilar but determined by the two-branch Siamese CNN model to be similar, margin is 1, and L 2 is a Euclidian distance;

the one or more non-transitory storage devices storing the computing instructions are further configured to run on the one or more processors and perform acts of:
combining the similarity between two digital images of the plurality of digital images with one or more content-based features and one or more feedback-bas

Assignees

• Wal Mart Stores Inc
• Walmart Apollo Llc

Inventors

• Guo Stephen Dean
• Achan Kannan
• Rapaka Venkata Syam Prakash

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F16/24578

  using ranking · CPC title

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• G06Q30/0631 Primary

  Recommending goods or services · CPC title