Systems and methods for joint learning of complex visual inspection tasks using computer vision

Patent metadata

Field	Value
Publication number	US-11508050-B2
Application number	US-201916721501-A
Country	US
Kind code	B2
Filing date	Dec 19, 2019
Priority date	Dec 19, 2018
Publication date	Nov 22, 2022
Grant date	Nov 22, 2022

Abstract

A method for performing automatic visual inspection includes: capturing visual information of an object using a scanning system including a plurality of cameras; extracting, by a computing system including a processor and memory, one or more feature maps from the visual information using one or more feature extractors; classifying, by the computing system, the object by supplying the one or more feature maps to a complex classifier to compute a classification of the object, the complex classifier including: a plurality of simple classifiers, each simple classifier of the plurality of simple classifiers being configured to compute outputs representing a characteristic of the object; and one or more logical operators configured to combine the outputs of the simple classifiers to compute the classification of the object; and outputting, by the computing system, the classification of the object as a result of the automatic visual inspection.

First claim

What is claimed is:

1. A method for performing automatic visual inspection, comprising:
capturing visual information of an object using a scanning system comprising a plurality of cameras;
extracting, by a computing system comprising a processor and memory, one or more feature maps from the visual information using one or more feature extractors;
classifying, by the computing system, the object by supplying the one or more feature maps to a complex classifier to compute a classification of the object, the complex classifier comprising:
a plurality of simple classifiers, each simple classifier of the plurality of simple classifiers being configured to compute outputs representing a characteristic of the object, and
one or more logical operators configured to combine the outputs of the plurality of simple classifiers to compute the classification of the object;

outputting, by the computing system, the classification of the object as a result of the automatic visual inspection;
wherein each simple classifier of the plurality of simple classifiers is configured by a corresponding threshold parameter of a plurality of threshold parameters, wherein the plurality of threshold parameters are jointly trained; and
wherein the plurality of threshold parameters are jointly trained by:
sampling a parameter space to select a plurality of sets of threshold parameters to configure the plurality of simple classifiers,
computing a True Positive rate (TPr) and a False Positive rate (FPr) for each set of threshold parameters of the plurality of sets of threshold parameters by:
configuring the complex classifier by configuring the plurality of simple classifiers based on the set of threshold parameters; and
computing the TPr and the FPr for the configuration by supplying the configured complex classifier with a validation set of data,

identifying a Pareto front comprising best performing sets of configuration parameters in accordance with the TPr and FPr for each set of the sets of configuration parameters, and
selecting a set of configuration parameters from the Pareto front in accordance with a rule set in accordance with a domain.

2. The method of claim 1 , wherein the one or more feature extractors comprise one or more convolutional neural networks.

3. The method of claim 1 , wherein the plurality of simple classifiers comprises one or more neural networks.

4. The method of claim 3 , wherein the plurality of simple classifiers comprises one or more support vector machines, and
wherein at least one logical operation is configured to combine an output of the one or more neural networks and an output of the one or more support vector machines.

5. The method of claim 1 , wherein the plurality of simple classifiers comprises one or more regression model.

6. The method of claim 1 , wherein the plurality of simple classifiers comprises one or more label-based classifier configured to perform on text detection.

7. The method of claim 1 wherein the visual information comprises color images, grayscale images, or depth maps.

8. The method of claim 7 , wherein the visual information comprises at least one depth map,
wherein the at least one depth map is captured by a depth camera system of the plurality of cameras.

9. The method of claim 8 , wherein the depth camera system comprises:
a time-of-flight depth camera;
a structured light depth camera;
a stereo depth camera comprising:
at least two color cameras;

a stereo depth camera comprising:
at least two color cameras; and
a color projector;

a stereo depth camera comprising:
at least two infrared cameras; or

a stereo depth camera comprising:
at least two infrared cameras;
an infrared projector; and
a color camera.

10. The method of claim 8 , wherein the plurality of simple classifiers comprises a classifier based on mathematical modeling of the at least one depth map.

11. The method of claim 1 , wherein a feature map of the one or more feature maps is provided as input to at least two of the plurality of simple classifiers.

12. The method of claim 1 , wherein the classification of the object comprises an identification of a category of a plurality of categories of objects.

13. The method of claim 1 , wherein the classification of the object comprises an identification of one or more properties of the object based on the visual information.

14. A visual inspection system comprising:
a scanner system comprising a plurality of cameras; and
a computing system connected to the scanner system over a computer network, the computing system comprising a processor and memory storing instructions that, when executed by the processor, cause the processor to:
control the scanner system to capture visual information of an object;
extract one or more feature maps from the visual information using one or more feature extractors;
classify, by the computing system, the object by supplying the one or more feature maps to a complex classifier to compute a classification of the object, the complex classifier comprising:
a plurality of simple classifiers, each simple classifier of the plurality of simple classifiers being configured to compute outputs representing a characteristic of the object, and
one or more logical operators configured to combine the outputs of the plurality of simple classifiers to compute the classification of the object;

output, by the computing system, the classification of the object as a result of an automatic visual inspection of the object;
wherein each simple classifier of the plurality of simple classifiers is configured by a corresponding threshold parameter of a plurality of threshold parameters, wherein the plurality of threshold parameters are jointly trained; and
wherein the plurality of threshold parameters are jointly trained by:
sampling a parameter space to select a plurality of sets of threshold parameters to configure the plurality of simple classifiers,
computing a True Positive rate (TPr) and a False Positive rate (FPr) for each set of threshold parameters of the plurality of sets of threshold parameters by:
configuring the complex classifier by configuring the plurality of simple classifiers based on the set of threshold parameters; and
computing the TPr and the FPr for the configuration by supplying the configured complex classifier with a validation set of data,

identifying a Pareto front comprising best performing sets of configuration parameters in accordance with the TPr and FPr for each set of the sets of configuration parameters, and
selecting a set of configuration parameters from the Pareto front in accordance with a rule set in accordance with a domain.

15. The visual inspection system of claim 14 , wherein the scanner system comprises at least one color camera.

16. The visual inspection system of claim 14 , wherein the scanner system comprises at least one depth camera.

17. The visual inspection system of claim 14 , further comprising a user device comprising a display device, the user device being configured to display:
the classification of the object; and
at least one characteristic of the object computed by at least one simple classifier of the plurality of simple classifiers.

18. The visual inspection system of claim 14 , wherein the computing system is configured to control a conveyor system to redirect movement of the object in accordance with the classification.

Assignees

• Packsize Llc

Inventors

• Dal Mutto Carlo
• Peruch Francesco
• Ou Alexander
• Hayes Robert

Classifications

• G06V10/765

  using rules for classification or partitioning the feature space · CPC title

• G06N20/10 Primary

  using kernel methods, e.g. support vector machines [SVM] · CPC title

• G06V20/80

  Recognising image objects characterised by unique random patterns · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T7/0002 Primary

  Inspection of images, e.g. flaw detection · CPC title