Reconstructor and contrastor for anomaly detection

Patent metadata

Field	Value
Publication number	US-10733722-B2
Application number	US-201815983342-A
Country	US
Kind code	B2
Filing date	May 18, 2018
Priority date	Jun 27, 2017
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

Systems and methods for detecting and correcting defective products include capturing at least one image of a product with at least one image sensor to generate an original image of the product. An encoder encodes portions of an image extracted from the original image to generate feature space vectors. A decoder decodes the feature space vectors to reconstruct the portions of the image into reconstructed portions by predicting defect-free structural features in each of the portions according to hidden layers trained to predict defect-free products. Each of the reconstructed portions are merged into a reconstructed image of a defect-free representation of the product. The reconstructed image is communicated to a contrastor to detect anomalies indicating defects in the product.

First claim

What is claimed is:

1. A method for detecting and correcting defective products, the method comprising: capturing at least one image of a product with at least one image sensor to generate an original defect-free image of the product; training a model only with the original defect-free image of the product as training data; encoding, with an encoder, each of at least one partly masked portion of an image extracted from the original defect-free image to generate a feature space vector; decoding, with a decoder, the feature space vector to reconstruct the at least one masked portion of the image into a corresponding at least one reconstructed portion by predicting defect-free structural features in each of the at least one masked portion according to hidden layers trained to predict defect-free products; merging each of the at least one reconstructed portion into a reconstructed image defining a defect-free representation of the product; communicating the defect-free representation of the reconstructed image to a contrastor to detect anomalies indicating defects in the product; and contrasting, with the contrastor, the original image with the defect-free representation of the reconstructed image to generate an anomaly map indicating shapes of the defects and locations of difference between the original image and the defect-free representation of the reconstructed image, wherein the encoder transforms the at least one partly masked portion of the image into a reduced dimension vector of numbers to obfuscate any features in unmasked regions of image patches and to reduce a risk of anomalies being present in a reconstructed representation of the image patches.

2. The method as recited in claim 1 , wherein the contrastor determines a pixel-by-pixel difference between the defect-free representation of the reconstructed image and the original image.

3. The method as recited in claim 1 , wherein the anomaly map includes a matrix of difference values corresponding to differences between the defect-free representation of the reconstructed image and the original image at a plurality of locations on the original image.

4. The method as recited in claim 1 , wherein the anomaly map includes an image depicting a difference between the defect-free representation of the reconstructed image and the original image at a plurality of locations on the original image.

5. The method as recited in claim 1 , further including tagging locations of difference as anomalies corresponding to defects on the product.

6. The method as recited in claim 1 , further including automatically discarding the product according to the detected anomalies.

7. The method as recited in claim 1 , further including an image patch extractor to extract at least one image patch and partially mask each of the at least one image patch to generate the at least one masked portion of the original defect-free image.

8. The method as recited in claim 7 , wherein the image patch extractor imposes a grid across the original defect-free image, with each area defined by the grid corresponds to a portion to be extracted.

9. The method as recited in claim 1 , further including notifying an operator of the anomalies.

10. The method as recited in claim 9 , further comprising a display for displaying the original image of the product with tags corresponding to the anomalies.

11. A method for detecting and correcting defective products, the method comprising: extracting, with an image patch extractor, portions of an original image of a product and partially masking each portion to form at least one masked portion; reconstructing, with a reconstructor, the original defect-free image, including: training a model only with the original defect-free image of the product as training data; encoding, with an encoder, each of at least one partly masked portion of an image extracted from the original defect-free image to generate a feature space vector; decoding, with a decoder, the feature space vector to reconstruct the at least one masked portion of the image into a corresponding at least one reconstructed portion by predicting defect-free structural features in each of the at least one masked portion according to hidden layers trained to predict defect-free products; merging the at least one reconstructed portion into a reconstructed image defining a defect-free representation of the product; contrasting, with a contrastor, the original image with the defect-free representation of the reconstructed image to generate an anomaly map indicating shapes of the defects and locations of difference between the original image and the defect-free representation of the reconstructed image; tagging the locations of difference as anomalies corresponding to defects on the product; and notifying, automatically via a display, an operator of the anomalies, wherein the encoder transforms the at least one partly masked portion of the image into a reduced dimension vector of numbers to obfuscate any features in unmasked regions of image patches and to reduce a risk of anomalies being present in a reconstructed representation of the image patches.

12. The method as recited in claim 11 , wherein the image patch extractor imposes a grid across the original defect-free image, where each area defined by the grid corresponds to a portion to be extracted.

13. The method as recited in claim 11 , further including blacking out an area of the portions to be reconstructed to generate the masked portions.

14. The method as recited in claim 11 , wherein the contractor determines a pixel-by-pixel difference between the defect-free representation of the reconstructed image and the original defect-free image.

15. The method as recited in claim 11 , wherein the display displays the original defect-free image of the product with tags corresponding to the anomalies.

16. A system for detecting and correcting defective products, the system including: at least one image sensor for capturing at least one image of a product to generate an original image of the product; a reconstructor to reconstruct an original defect-free image of a product by reconstructing partially masked portions of the original defect-free image to be a defectless representation of corresponding portions of the product, the reconstructor including: a model trained only with the original defect-free image of the product as training data; an encoder to encode each of at least one partially masked portion of an image extracted from the original defect-free image to generate a feature space vector; a decoder to decode the feature space vector to reconstruct the at least one portion of the image into a corresponding at least one reconstructed portion by predicting defect-free structural features in each of the at least one portion according to hidden layers trained to predict defect-free products; an image merger to merge each of the at least one reconstructed portion into a reconstructed image defining a defect-free representation of the product; a contrastor for receiving the defect-free representation of the reconstructed image to detect anomalies indicating defects in the product, wherein the contrastor contrasts the original image with the defect-free representation of the reconstructed image to generate an anomaly map indicating shapes of the defects and locations of difference between the original image and the defect-free representation of the reconstructed image; and an anomaly correction system to automatically discard the product according to tagged anomalies, wherein the encoder transforms the at least one partly masked portion of the image into a reduced dimension

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Niculescu-Mizil Alexandru
• Cosatto Eric
• Wu Felix

Classifications

• G06T7/0008 Primary

  checking presence/absence · CPC title

• G06T7/0004 Primary

  Industrial image inspection · CPC title

• G06T12/30

  Image post-processing, e.g. metal artefact correction · CPC title

• G06T11/26

  Drawing of charts or graphs · CPC title

• G06F18/2433

  Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection · CPC title