Systems and methods for augmented reality visual inspections

Patent metadata

Field	Value
Publication number	US-11836865-B2
Application number	US-202117448005-A
Country	US
Kind code	B2
Filing date	Sep 17, 2021
Priority date	Sep 18, 2020
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

An augmented reality visualization system is disclosed for viewing non-destructive inspection (NDI) data of a component. The system comprises a component camera configured to capture a real time image of the component, a component display configured to display an AR image of the component, and an AR controller including an augmented reality processor and a non-transitory tangible storage medium. The storage medium includes processor-executable instructions for controlling the AR processor when the instructions are executed by the AR processor. The instructions access the NDI data of the component and implement a component AR engine to generate the AR image in real time by overlaying the NDI data on the real time image. The AR engine is configured to detect one or more three-dimensional (3D) features of the component in the real-time image and generate the AR image based on the detected one or more features.

First claim

What is claimed is:

1. An augmented reality visualization system for viewing non-destructive inspection data of a component, the system comprising:
a component camera configured to capture a real time image of the component;
a component display configured to display an augmented reality image of the component;
a non-destructive inspection database storing the non-destructive inspection data and a 3D computer model representative of the component, the non-destructive inspection data being mapped to a 3D computer model in the non-destructive inspection database; and
an augmented reality controller including an augmented reality processor and a non-transitory tangible storage medium including processor-executable instructions for controlling the augmented reality processor when the instructions are executed by the augmented reality processor, the instructions including instructions to access the non-destructive inspection data of the component and to implement a component augmented reality engine to generate the augmented reality image in real time by overlaying the non-destructive inspection data on the real time image, the augmented reality engine being configured to detect one or more three-dimensional (3D) features of the component in the real-time image, match the one or more 3D features to the 3D computer model to orient and align the non-destructive inspection data in relation to the real time image, and generate the augmented reality image such that the augmented reality image includes the non-destructive inspection data oriented and aligned on the real time image based on the matching of the detected one or more 3D features to the 3D computer model.

2. The augmented reality visualization system as set forth in claim 1 , wherein the component augmented reality engine is configured to determine a displayed portion of the component in the real time image and to overlay only the non-destructive inspection data for the displayed portion.

3. The augmented reality visualization system as set forth in claim 2 , wherein the component augmented reality engine is configured to determine, based on the detected one or more 3D features, which of the non-destructive inspection data corresponds to the displayed portion.

4. The augmented reality visualization system as set forth in claim 3 , wherein the component augmented reality engine is configured to select and transform the non-destructive inspection data corresponding to the displayed portion by comparing the 3D computer model to the displayed portion before overlaying the non-destructive inspection data on the displayed portion to create the augmented reality image.

5. The augmented reality visualization system as set forth in claim 1 , further comprising a machine-readable identification marking visibly associated with the component, and wherein the instructions to access the non-destructive inspection data include instructions to read the machine-readable identification marking in the real time image and to determine a non-destructive inspection record for the component based on the machine-readable identification marking in the real time image, the non-destructive inspection record including the non-destructive inspection data of the component.

6. The augmented reality visualization system as set forth in claim 5 , further comprising a non-destructive inspection database storing a plurality of non-destructive inspection records for a plurality of different components, including the non-destructive inspection record for the component.

7. The augmented reality visualization system as set forth in claim 6 , further comprising a remote server storing the non-destructive inspection database, the augmented reality controller communicatively coupled to the remote server.

8. The augmented reality visualization system as set forth in claim 1 , wherein the non-destructive inspection data includes a plurality of types of non-destructive inspection data for the component.

9. The augmented reality visualization system as set forth in claim 8 , further comprising a user interface including a user input and a display, the user interface including a non-destructive inspection data selector configured to receive user inputs selecting the type of non-destructive inspection data for the component that is displayed in the augmented reality image.

10. The augmented reality visualization system as set forth in claim 8 , wherein the types of non-destructive inspection data include at least two of thermography data, ultrasonic data, and shearography data.

11. A computer implemented method for viewing non-destructive testing inspection data of a component in real time, the method comprising;
mapping the non-destructive inspection data to a 3D computer model of the component containing a plurality of 3D features;
capturing, by a component camera, a real time image of the component; accessing the non-destructive inspection data of the component;
detecting one or more three-dimensional (3D) features of the component in the real-time image, matching the one or more 3D features in the real time image to the 3D features in the 3D computer model of the component, and using the one or more 3D features as one or more reference points for orienting and aligning the non-destructive inspection data in relation to the real time image;
generating an augmented reality image based on the detected one or more 3D features in real time by overlaying the non-destructive inspection data on the real time image such that the augmented reality image includes the non-destructive inspection data oriented and aligned on the real time image based on said matching the one or more 3D features in the real time image to the 3D features in the 3D computer model of the component; and
displaying, by a component display, the augmented reality image.

12. The method of claim 11 , wherein generating the augmented reality image includes determining a displayed portion of the component in the real time image and overlaying the non-destructive inspection data for the displayed portion.

13. The method of claim 12 , wherein generating the augmented reality image includes determining, based on the detected one or more 3D features, which of the non-destructive inspection data corresponds to the displayed portion.

14. The method of claim 13 , wherein generating the augmented reality image includes selecting and transforming the non-destructive inspection data corresponding to the displayed portion by comparing spatial information representative of one or more spatial features of the component to the displayed portion and overlaying the selected and scaled non-destructive inspection data on the displayed portion.

15. The method of claim 11 , wherein accessing the non-destructive inspection data includes reading a machine-readable identification marking in the real time image and retrieving the non-destructive inspection data from a non-destructive inspection database based on the machine-readable identification markings in the real time image.

16. The method of claim 11 , further comprising receiving a selection of at least one type of non-destructive inspection data from among a plurality of types of non-destructive inspection data stored in a non-destructive inspection database, and in response the selection, displaying the at least one type of non-destructive inspection data on the component display.

17. The method of claim 16 , wherein the types of non-destructive inspection data include at least two of thermography data, ultrasonic data, and shearography data.

18. The method of claim

Assignees

• Univ Wichita State

Inventors

• Seneviratne Waruna
• Tomblin John
• Pini Christopher

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/14

  Digital output to display device {; Cooperation and interconnection of the display device with other functional units} · CPC title

• G06F16/22

  Indexing; Data structures therefor; Storage structures · CPC title

• G06T7/33

  using feature-based methods · CPC title

• G06F16/9038 Primary

  Presentation of query results · CPC title