System and method for produce detection and classification

Patent metadata

Field	Value
Publication number	US-11734813-B2
Application number	US-202217867922-A
Country	US
Kind code	B2
Filing date	Jul 19, 2022
Priority date	Jul 26, 2018
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

Systems, methods, and computer-readable storage media for object detection and classification, and particularly produce detection and classification. A system configured according to this disclosure can receiving, at a processor, an image of an item. The system can then perform, across multiple pre-trained neural networks, feature detection on the image, resulting in feature maps of the image. These feature maps can be concatenated and combined, then input into an additional neural network for feature detection on the combined feature map, resulting in tiered neural network features. The system then classifies, via the processor, the item based on the tiered neural network features.

First claim

We claim:

1. A method comprising:
receiving, at a processor, an image of an item;
performing, via the processor using a first pre-trained neural network, feature detection on the image, resulting in a first feature map of the image;
performing, via the processor using a second pre-trained neural network, feature detection on the image, resulting in a second feature map of the image;
creating, via the processor, a combined feature map based on the first feature map and the second feature map;
performing, via the processor using a third pre-trained neural network, feature detection on the combined feature map, resulting in tiered neural network features; and
classifying, via the processor, the item based on the tiered neural network features, the classifying including implementing a set of pre-trained neural networks, the set of pre-trained neural networks having been produced based on the tiered neural network features, the classification being a combination of results of the set of pre-trained neural networks, and a result of each pre-trained neural network of the set of pre-trained neural networks being weighted based on a corresponding accuracy.

2. The method of claim 1 , wherein the item is produce.

3. The method of claim 2 , wherein the feature detection identifies defects within the produce.

4. The method of claim 1 , wherein at least one of the first pre-trained neural network, the second pre-trained neural network, and the third pre-trained neural network is a Faster Regional Convolutional Neural Network.

5. The method of claim 4 , wherein the Faster Regional Convolutional Neural Network identifies a top-left coordinate of a rectangular region for each item within the image and a bottom-right coordinate of the rectangular region.

6. The method of claim 1 , wherein the third pre-trained neural network uses distinct neural links than the neural links of the first pre-trained neural network and the second pre-trained neural network.

7. The method of claim 1 , wherein the processor is a Graphical Processing Unit.

8. A system, comprising:
a processor; and a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising:
receiving an image of an item;
performing, using a first pre-trained neural network, feature detection on the image, resulting in a first feature map of the image;
performing, using a second pre-trained neural network, feature detection on the image, resulting in a second feature map of the image;
creating, via the processor, a combined feature map based on the first feature map and the second feature map;
performing, using a third pre-trained neural network, feature detection on the combined feature map, resulting in tiered neural network features; and
classifying the item based on the tiered neural network features, the classifying including implementing a set of pre-trained neural networks, the set of pre-trained neural networks having been produced based on the tiered neural network features, the classification being a combination of results of the set of pre-trained neural networks, and a result of each pre-trained neural network of the set of pre-trained neural networks being weighted based on a corresponding accuracy.

9. The system of claim 8 , wherein the item is produce.

10. The system of claim 9 , wherein the feature detection identifies defects within the produce.

11. The system of claim 8 , wherein at least one of the first pre-trained neural network, the second pre-trained neural network, and the third pre-trained neural network is a Faster Regional Convolutional Neural Network.

12. The system of claim 11 , wherein the Faster Regional Convolutional Neural Network identifies a top-left coordinate of a rectangular region for each item within the image and a bottom-right coordinate of the rectangular region.

13. The system of claim 8 , wherein the third pre-trained neural network uses distinct neural links than the neural links of the first pre-trained neural network and the second pre-trained neural network.

14. The system of claim 8 , wherein the processor is a Graphical Processing Unit.

15. A non-transitory computer-readable storage medium having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:
receiving an image of an item;
performing, using a first pre-trained neural network, feature detection on the image, resulting in a first feature map of the image;
performing, using a second pre-trained neural network, feature detection on the image, resulting in a second feature map of the image;
creating, via the processor, a combined feature map based on the first feature map and the second feature map;
performing, using a third pre-trained neural network, feature detection on the combined feature map, resulting in tiered neural network features; and
classifying the item based on the tiered neural network features, the classifying including implementing a set of pre-trained neural networks, the set of pre-trained neural networks having been produced based on the tiered neural network features, the classification being a combination of results of the set of pre-trained neural networks, and a result of each pre-trained neural network of the set of pre-trained neural networks being weighted based on a corresponding accuracy.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the item is produce.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the feature detection identifies defects within the produce.

18. The non-transitory computer-readable storage medium of claim 15 , wherein at least one of the first pre-trained neural network, the second pre-trained neural network, and the third pre-trained neural network is a Faster Regional Convolutional Neural Network.

19. The non-transitory computer-readable storage medium of claim 18 , wherein the Faster Regional Convolutional Neural Network identifies a top-left coordinate of a rectangular region for each item within the image and a bottom-right coordinate of the rectangular region.

20. The non-transitory computer-readable storage medium of claim 15 , wherein the third pre-trained neural network uses distinct neural links than the neural links of the first pre-trained neural network and the second pre-trained neural network.

Assignees

• Walmart Apollo Llc

Inventors

• Mathew Issac
• Pushp Pushkar
• Patel Viraj
• Xavier Emily
• Savlani Gaurav
• Nellore Venkataraja
• Agarwal Rahul
• Thiruvenkadam Girish
• Naik Shivani

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T7/0004 Primary

  Industrial image inspection · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06N3/045

  Combinations of networks · CPC title