Mobile visual-inspection system

Patent metadata

Field	Value
Publication number	US-11238675-B2
Application number	US-201815945640-A
Country	US
Kind code	B2
Filing date	Apr 4, 2018
Priority date	Apr 4, 2018
Publication date	Feb 1, 2022
Grant date	Feb 1, 2022

Abstract

A ground-based visual-inspection system includes a ground-based visual-inspection apparatus and a control system. The ground-based visual-inspection apparatus includes a mobile base, an actuatable arm coupled to the mobile base, and an effector coupled to the actuatable arm. The actuatable arm is locatable in a three dimensional space. The end effector includes a camera configured to capture images of a structure, such as an aircraft. The control system is configured to determine location information of the camera relative to a reference location and associate the location information with the images.

First claim

What is claimed is:

1. A ground-based visual-inspection system for inspecting a structure, the ground-based visual-inspection system comprising:
a ground-based visual-inspection apparatus comprising:
a mobile base that translationally moves relative to a reference position to define a translational displacement of the mobile base;
an actuatable arm coupled to the mobile base, the actuatable arm locatable in a three-dimensional space; and
an end effector coupled to the actuatable arm, the end effector comprising:
a set of at least three laser rangefinders; and
a camera, wherein the camera is oriented relative to the structure in response to input from the set of at least three rangefinders, wherein a perpendicular distance between the structure and the camera is measured by the rangefinders, wherein a rotational orientation of the camera is determined relative to the reference position of the mobile base, and wherein the camera is configured to capture a set of visual images of a surface of the structure; and

a control system that embeds the perpendicular distance to the surface of the structure within the set of visual images captured by the camera, the control system comprising:
an alignment system comprising the set of at least three laser rangefinders spaced around a perimeter of the camera; and
a feedback system configured to:
utilize the set of at least three laser rangefinders to measure the perpendicular distance and the rotational orientation of the camera relative to each target area located on the surface of the structure without using optical targets, wherein the camera and the set of at least three laser rangefinders are non-movably fixed, relative to each other, and the end effector rotates the set of at least three laser rangefinders relative to the structure and moves the camera toward or away from the structure until the set of at least three laser rangefinders measures the same perpendicular distance to the surface of the structure;
utilize the feedback system to, while translationally moving the camera along the surface of the structure:
automatically adjust the perpendicular distance and the rotational orientation of the camera to maintain the set of at least three laser rangefinders at the same perpendicular distance from the surface of the structure and to maintain the rotational orientation of the camera perpendicular to the surface of the structure being captured by the set of visual images as the ground-based visual-inspection apparatus translationally moves the end effector relative to the reference position; and
continuously adjust the alignment of the camera to maintain the perpendicular alignment of the camera relative to each target area located on the surface of the structure as the mobile base moves along the ground beside the structure and the actuatable arm moves in three-dimensions relative to the surface of the structure during inspection of the structure,

determine location information of the camera in three-dimensions relative to a reference location, and
associate the three-dimensional location information of the camera with location information for each respective target area located on the surface of the structure captured by the set of visual images as the mobile base moves along the ground beside the structure and the actuatable arm moves in three-dimensions relative to the surface of the structure during the inspection of the structure.

2. The ground-based visual-inspection system according to claim 1 , wherein the control system is further configured to determine the location information by determining a location of the camera relative to the reference location by:
acquiring a base location by computing a transformation matrix of the mobile base relative to the reference location;
acquiring an arm location by computing a transformation matrix of the actuatable arm relative to the reference location; and
performing matrix multiplication of the transformation matrix of the mobile base and the transformation matrix of the actuatable arm to compute a transformation matrix of the end effector relative to the reference location.

3. The ground-based visual-inspection system according to claim 1 , wherein the control system is further configured to embed orientation information of the camera in the set of visual images.

4. The ground-based visual-inspection system according to claim 1 , wherein the control system is configured to embed the location information into the set of visual images.

5. The ground-based visual-inspection system according to claim 4 , wherein the location information is based on data from sensors on the ground-based visual-inspection apparatus or the structure.

6. The ground-based visual-inspection system according to claim 1 , wherein the location information is based on the reference location on the structure.

7. The ground-based visual-inspection system according to claim 1 , wherein the mobile base comprises a cart maneuverable by manual power.

8. The ground-based visual-inspection system according to claim 1 , wherein:
the control system is further configured to embed orientation information within the set of visual images; and
the control system is further configured to embed the location information into the set of visual images.

9. The ground-based visual-inspection system according to claim 8 , wherein the location information comprises the translational displacement from the reference position and rotational displacement from a reference orientation of the mobile base relative to the reference location.

10. The ground-based visual-inspection system according to claim 1 , wherein the actuatable arm comprises an indexing rail.

11. A method comprising:
aligning a ground-based visual-inspection apparatus to a structure, the ground-based visual-inspection apparatus comprising a mobile base that translates relative to a reference position, an actuatable arm coupled to the mobile base, and an end effector coupled to the actuatable arm, wherein the end effector comprises a camera oriented relative to a structure wherein a perpendicular distance between the camera and the structure is measured by an alignment system comprising a set of at least three laser rangefinders spaced around a perimeter of the camera;
measuring, via the set of at least three laser range finders, a perpendicular distance between the camera and the structure and an orientation of the camera relative to each target area located on a surface of the structure without using optical targets;
automatically adjusting the set of at least three laser rangefinders so that they each measure the same perpendicular distance from the surface of the structure to the camera and adjusting an orientation of the camera that is perpendicular to the surface of the structure being captured by the set of visual images as the ground-based visual-inspection apparatus changes position;
continuously adjusting the alignment of the camera to maintain the perpendicular alignment of the camera relative to each target area located on the surface of the structure as the mobile base moves along the ground beside the structure and the actuatable arm moves in three-dimensions relative to the surface of the structure during inspection of the structure;
capturing a set of visual images of each target area on the surface of the structure with the camera; wherein the perpendicular distance between the camera and the surface of the structure is embedded in the set of visual images;
determining location information of the camera in three-dimensions relative to a reference location by acquiring a base location, arm location, and end effector location relative to a reference location

Assignees

• Boeing Co

Inventors

• Troy James J
• Lea Scott W
• Wright Daniel J
• Georgeson Gary E

Classifications

• G07C5/0808 Primary

  Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005) · CPC title

• B64F5/60 Primary

  Testing or inspecting aircraft components or systems · CPC title

• H04N23/69

  Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming · CPC title

• G05B2219/40298

  Manipulator on vehicle, wheels, mobile · CPC title

• B25J9/1697

  Vision controlled systems · CPC title