Synthetic image generation for surface anomaly detection

Patent metadata

Field	Value
Publication number	US-11599986-B2
Application number	US-202016902588-A
Country	US
Kind code	B2
Filing date	Jun 16, 2020
Priority date	Jun 16, 2020
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

Systems and methods for detecting surface anomalies are disclosed. For example, a computer-implemented method for detecting surface anomalies on an object comprises receiving measured electromagnetic radiation (EMR) profiles for the object, generating synthetic EMR profiles for the object based on the measured EMR profiles, determining whether the object contains a surface anomaly based on the measured EMR profiles and the synthetic EMR profiles, and indicating a surface anomaly to a user via a display when a surface anomaly is detected. In another example, a system comprises a computing device comprising non-transitory memory with computer-readable instructions for receiving unpaired image data of an object of two different image types, predicting missing image data to generate paired image data of the two different image types, and determining whether the object contains a surface anomaly based on the paired image data. The computing device comprises a processor configured to execute the computer-readable instructions.

First claim

The invention claimed is:

1. A computer implemented method for detecting surface anomalies on an object, the method comprising:
receiving two or more measured electromagnetic radiation (EMR) profiles for the object, wherein at least one EMR profile of the two or more measured EMR profiles is captured by one or more EMR receivers at a first set of ambient conditions, and wherein at least one other EMR profile of the two or more measured EMR profiles is captured by at least one of the one or more EMR receivers at a second set of ambient conditions, wherein the second set of ambient conditions is different than the first set of ambient conditions in one or more of ambient temperature, ambient humidity, ambient pressure, or ambient precipitation condition;
generating, with a trained image translation model, two or more synthetic EMR profiles for the object based on the two or more measured EMR profiles;
determining whether the object contains a surface anomaly based on the two or more measured EMR profiles, the two or more synthetic EMR profiles, the first set of ambient conditions, and the second set of ambient conditions; and
responsive to the determining that the object contains a surface anomaly, indicating the surface anomaly to a user via a display unit.

2. The method of claim 1 , wherein the first set of ambient conditions and the second set of ambient conditions further are different n ambient lighting condition.

3. The method of claim 2 , wherein the first set of ambient conditions and the second set of ambient conditions are different in light intensity, light orientation, light variation, light patterns, or light sequence.

4. The method of claim 1 , wherein the determining whether the object contains a surface anomaly is based on one or more physical characteristics of the object, wherein the one or more physical characteristics of the object comprise one or more of a shape, a color, a material composition, a surface coating, and one or more reflective properties of the object.

5. The method of claim 1 , wherein the determining whether the object contains a surface anomaly is based on a location of a region of interest of the object.

6. The method of claim 5 , wherein the object is an aircraft, and wherein the region of interest is a portion of the aircraft, and wherein the determining whether the object contains a surface anomaly based on the location of the region of interest is based on one or more of one or more parts included in the region of interest, a composition of the one or more parts, and a surface coating of the one or more parts.

7. The method of claim 1 , wherein, together or separately, the two or more measured EMR profiles and the two or more synthetic EMR profiles comprise different types of EMR profiles that represent different ranges of wavelengths.

8. The method of claim 7 , wherein the different types of EMR profiles comprise a first type of EMR profile that represents a first range of wavelengths and a second type of EMR profile that represents a second range of wavelengths, wherein the first range of wavelengths is different than the second range of wavelengths.

9. The method of claim 8 , wherein the generating the two or more synthetic EMR profiles for the object based on the two or more measured EMR profiles comprises one or more of converting the first type of EMR profile to the second type of EMR profile and converting the second type of EMR profile to the first type of EMR profile.

10. The method of claim 1 , further comprising training an untrained image translation model to generate the trained image translation model.

11. The method of claim 10 , wherein the training the untrained image translation model comprises binning two or more training EMR profiles based on one or more of one or more ambient conditions, a location of a region of interest of the object, and one or more physical characteristics of the object.

12. The method of claim 1 , wherein the determining whether the object contains a surface anomaly based on the two or more measured EMR profiles and the two or more synthetic EMR profiles comprises comparing the two or more measured EMR profiles and the two or more synthetic EMR profiles to two or more control EMR profiles of an anomaly-free object.

13. A method for detecting surface anomalies on an object, the method comprising:
receiving two or more images of the object taken at different ambient conditions by one or more cameras, the two or more images comprising one or more of two different image types, wherein the different ambient conditions comprise one or more of different ambient temperatures or different ambient humidities;
calculating, with a trained image translation model, for each of the two or more images, a paired image, wherein the paired image comprises the other of the two different image types; and
determining whether the object contains a surface anomaly based on the received two or more images, the calculated paired images, and the different ambient conditions.

14. The method of claim 13 , wherein the different ambient conditions further comprise different ambient lighting conditions.

15. The method of claim 13 , wherein the determining whether the object contains a surface anomaly is based on a location of a region of interest and one or more physical characteristics of the object at the region of interest.

16. The method of claim 13 , wherein the method further comprises creating the trained image translation model by training an untrained image translation model, wherein the training comprises binning training data based on one or more ambient conditions.

17. The method of claim 16 , wherein the trained image translation model comprises two or more sub-models, and wherein the training the untrained image translation model further comprises conditioning each of the two or more sub-models with different bins of training data, and wherein the calculating the paired image comprises selecting one of the two or more sub-models to perform the calculating based on the different ambient conditions at which the two or more images were taken and the one or more ambient conditions on which the untrained image translation model was trained.

18. A system comprising:
a computing device comprising:
non-transitory memory comprising computer-readable instructions for:
receiving unpaired image data of an object, the unpaired image data comprising two different image types representing two different wavelength ranges of electromagnetic radiation (EMR);
predicting missing image data to generate paired image data of the two different image types;
determining whether the object contains a surface anomaly based on the paired image data;
adjusting the predicted missing image data based on changes in one or more ambient conditions, wherein the one or more ambient conditions comprise one or more of ambient temperature, ambient lighting, and ambient humidity; and

a processing unit configured to execute the computer-readable instructions.

19. The system of claim 18 , further comprising a display configured to indicate to a user whether the object contains the surface anomaly, and wherein the display is configured to indicate one or more of a location of the surface anomaly, a type of the surface anomaly, and a morphology or geometry of the surface anomaly to the user.

20. The method of claim 10 , wherein the trained image translation model comprises two or more sub-models, and wherein the training the untrained image translation model further comprises binning training

Assignees

• Boeing Co

Inventors

• Afrasiabi Amir
• Kelsey William David

Classifications

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06T7/0004 Primary

  Industrial image inspection · CPC title

• G06T7/001 Primary

  using an image reference approach · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06T2207/30156

  Vehicle coating · CPC title