Systems and methods for product identification using image analysis and trained neural network

Patent metadata

Field	Value
Publication number	US-11294952-B2
Application number	US-202016947142-A
Country	US
Kind code	B2
Filing date	Jul 20, 2020
Priority date	Feb 6, 2020
Publication date	Apr 5, 2022
Grant date	Apr 5, 2022

Abstract

Disclosed are methods, systems, and non-transitory computer-readable medium for analysis of images including wearable items. For example, a method may include obtaining a first set of images, each of the first set of images depicting a product; obtaining a first set of labels associated with the first set of images; training an image segmentation neural network based on the first set of images and the first set of labels; obtaining a second set of images, each of the second set of images depicting a known product; obtaining a second set of labels associated with the second set of images; training an image classification neural network based on the second set of images and the second set of labels; receiving a query image depicting a product that is not yet identified; and performing image segmentation of the query image and identifying the product in the image by performing image analysis.

First claim

What is claimed is:

1. A computer-implemented method comprising:
obtaining, by one or more processors, a set of images, wherein each of the set of images depicts a product;
obtaining, by the one or more processors, a set of labels associated with the set of images, wherein each of the set of labels corresponds to an image in the set of images and includes information indicating a mask or classification of the product depicted in the corresponding image;
training, by the one or more processors, an image segmentation neural network based on the set of images and the set of labels;
receiving, by the one or more processors, a query image depicting a product that is not yet identified;
performing, by the one or more processors, image segmentation of the query image using the image segmentation neural network, wherein performing image segmentation of the query image includes removing background image portions of the query image, thereby obtaining a mask image of the product depicted in the query image; and
identifying the product in the image by performing, by the one or more processors, image analysis of the mask image of the product depicted in the query image using an image classification neural network.

2. The computer-implemented method of claim 1 , wherein the classification information comprises one or more categories of the product, one or more patterns of the product, one or more colors of the product, one or more sleeve lengths of the product, one or more hemline lengths of the product, one or more neckline shapes of the product, one or more pattern spatial frequencies associated with the product, one or more hues associated the product, one or more saturations associated with the product, and/or one or more lightness values associated with the product.

3. The computer-implemented method of claim 1 , further comprising:
testing, by the one or more processors, the image classification neural network based on the set of images and the set of labels.

4. The computer-implemented method of claim 1 , wherein performing image analysis of the mask image comprises:
classifying the product depicted in the query image, wherein classifying the product depicted in the query image includes determining one or more categories of the depicted product, determining one or more patterns of the depicted product, determining one or more colors of the depicted product, determining one or more sleeve lengths of the depicted product, determining one or more hemline lengths of the depicted product, determining one or more neckline shapes of the depicted product, determining one or more pattern spatial frequencies associated with the depicted product, determining one or more hues associated the depicted product, determining one or more saturations associated with the depicted product, and/or determining one or more lightness associated with the depicted product.

5. The computer-implemented method of claim 4 , further comprising:
generating one or more recommendations based on the identified product, wherein the one or more recommendations include one or more products associated with the classified product depicted in the query image.

6. The computer-implemented method of claim 5 , wherein generating the one or more recommendations based on the identified product comprises:
obtaining the one or more products based on the classified product depicted in the query image;
determining a similarity score for each of the one or more products; and
ranking the one or more products based on the similarity scores.

7. The computer-implemented method of claim 6 , further comprising displaying one or more of the ranked products.

8. A computer system comprising:
a data storage device storing processor-readable instructions; and
a processor configured to execute the instructions to perform a method including:
obtaining a set of images, wherein each of the set of images depicts a product;
obtaining a set of labels associated with the set of images, wherein each of the set of labels corresponds to an image in the set of images and includes information indicating a mask or classification of the product depicted in the corresponding image;
training an image segmentation neural network based on the set of images and the set of labels;
receiving a query image depicting a product that is not yet identified;
performing image segmentation of the query image using the image segmentation neural network, wherein performing image segmentation of the query image includes removing background image portions of the query image, thereby obtaining a mask image of the product depicted in the query image; and
identifying the product in the image by performing image analysis of the mask image of the product depicted in the query image using an image classification neural network.

9. The computer system of claim 8 , wherein the classification information comprises one or more categories of the product, one or more patterns of the product, one or more colors of the product, one or more sleeve lengths of the product, one or more hemline lengths of the product, one or more neckline shapes of the product, one or more pattern spatial frequencies associated with the product, one or more hues associated the product, one or more saturations associated with the product, and/or one or more lightness values associated with the product.

10. The computer system of claim 8 , wherein the method further includes:
testing the image classification neural network based on the set of images and the set of labels.

11. The computer system of claim 8 , wherein performing image analysis of the mask image comprises:
classifying the product depicted in the query image, wherein classifying the product depicted in the query image includes determining one or more categories of the depicted product, determining one or more patterns of the depicted product, determining one or more colors of the depicted product, determining one or more sleeve lengths of the depicted product, determining one or more hemline lengths of the depicted product, determining one or more neckline shapes of the depicted product, determining one or more pattern spatial frequencies associated with the depicted product, determining one or more hues associated the depicted product, determining one or more saturations associated with the depicted product, and/or determining one or more lightness values associated with the depicted product.

12. The computer system of claim 11 , the method further including:
generating one or more recommendations based on the identified product, wherein the one or more recommendations include one or more products associated with the classified product depicted in the query image.

13. The computer system of claim 12 , wherein generating the one or more recommendations based on the identified product comprises:
obtaining the one or more products based on the classified product depicted in the query image;
determining a similarity score for each of the one or more products; and
ranking the one or more products based on the similarity scores.

14. The computer system of claim 13 , further comprising displaying one or more of the ranked products.

15. A non-transitory computer-readable medium containing instructions that, when executed by a processor, cause the processor to perform a method comprising:
obtaining a set of images, wherein each of the set of images depicts a product;
obtaining a set of labels associated with the set of images, wherein each of the set of labels corresponds to an image in the set of images and includes information indicating a mask or classification of th

Assignees

• Caastle Inc

Inventors

• Tsai Yu-Cheng
• Jiang Dongming
• Goldenberg Georgiy

Classifications

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06Q30/0625 Primary

  by formulating product or service queries, e.g. using keywords or predefined options · CPC title

• G06F16/583 Primary

  using metadata automatically derived from the content · CPC title