Robotic system architecture and control processes

Patent metadata

Field	Value
Publication number	US-11504853-B2
Application number	US-201715814965-A
Country	US
Kind code	B2
Filing date	Nov 16, 2017
Priority date	Nov 16, 2017
Publication date	Nov 22, 2022
Grant date	Nov 22, 2022

Abstract

A system includes a first sensor having a fixed location relative to a workspace, a second sensor, at least one robotic manipulator coupled to a manipulation tool, and a control system in communication with the at least one robotic manipulator. The control system is configured to determine a location of a workpiece in the workspace based on first sensor data from the first sensor and a three-dimensional (3D) model corresponding to the workpiece. The control system is configured to map a set of 2D coordinates from a second 2D image from the second sensor to a set of 3D coordinates based on the location, and to generate one or more control signals for the at least one robotic manipulator based on the set of 3D coordinates.

First claim

What is claimed is:

1. A system, comprising:
a first sensor having a fixed location relative to a workspace;
a second sensor;
at least one robotic manipulator coupled to a manipulation tool and configured for movement in the workspace; and
a control system in communication with the at least one robotic manipulator, the control system configured to determine a location of a workpiece in the workspace based on sensor data from the first sensor and a three-dimensional (3D) model corresponding to the workpiece, the control system configured to map a set of two-dimensional (2D) coordinates from a 2D image of the workpiece from the second sensor to a set of 3D coordinates based on the location, the control system configured to generate one or more control signals for the at least one robotic manipulator to cause the manipulation tool to manipulate a surface of the workpiece based on the set of 3D coordinates;
wherein:
the control system is configured to identify from the 2D image an indication of a defect associated with the surface of the workpiece;
the set of 2D coordinates are 2D image coordinates from the 2D image that correspond to the indication of the defect;
the control system is configured to generate one or more tool paths on the surface of the workpiece for the manipulation tool to interact with the surface of the workpiece based on grouping 3D coordinates of the set according to a size of the manipulation tool, the grouping being accomplished by creating from the set of 3D coordinates a plurality of point clusters, each point cluster including a subset of the set of 3D coordinates that is within a common region of the workpiece having a size based on the size of the manipulation tool;
the control system is configured to generate the one or more control signals based on the one or more tool paths on the surface of the workpiece and wherein crossover between different tool paths on the surface of the workpiece is controlled to have the manipulation tool contact the indication of the defect a minimal number of times and to simultaneously obtain full coverage of an entire area of the indication of the defect by the manipulation tool.

2. The system of claim 1 , wherein:
the at least one robotic manipulator includes a first robotic manipulator and a second robotic manipulator;
the first robotic manipulator is coupled to the manipulation tool;
the second robotic manipulator is coupled to the second sensor; and
the second sensor is an ultraviolet (UV) image capture device configured to generate the 2D image of the workpiece.

3. The system of claim 2 , wherein:
the 2D image is a second 2D image;
the sensor data includes a first 2D image;
the manipulation tool is a brush;
the set of 2D coordinates are 2D image coordinates from the second 2D image that correspond to the indication of the defect; and
the indication of the defect is an indication associated with fluorescent penetrant.

4. The system of claim 1 , wherein:
the control system is configured to generate a projection matrix based on projecting a predefined three-dimensional (3D) model of the workpiece onto sensor data including a first 2D image including at least a portion of the workpiece, the projection matrix includes a mapping between 2D image coordinates and the 3D coordinates corresponding to the 3D model; and
the control system is configured to map the set of 2D coordinates to the set of 3D coordinates based on the projection matrix.

5. The system of claim 4 , wherein the control system is configured to generate the projection matrix by:
creating a virtual 2D image of the 3D model after projecting the 3D model onto the first 2D image;
comparing the virtual 2D image of the 3D model to the first 2D image to determine one or more deviations of the workpiece from the 3D model; and
generating the mapping between 2D image coordinates and 3D coordinates based on the one or more deviations between the workpiece and the 3D model.

6. The system of claim 1 , wherein the control system is configured to create the plurality of point clusters by:
defining a virtual camera having a field of view based on a diameter of the manipulation tool;
determining if all 3D coordinates of the set are within the field of view of the virtual camera or the field of view of any previously defined virtual cameras;
repeating the defining and determining until all 3D coordinates of the set are within the field of view of at least one virtual camera; and
assigning to each point cluster a corresponding subset of 3D coordinates that are within the field of view of the virtual camera associated with the point cluster;
wherein each 3D coordinate is associated with a single point cluster.

7. The system of claim 6 , wherein the control system is configured to:
generate a graph to connect the plurality of point clusters based on one or more constraints; and
selectively generate a link between individual point clusters based on a distance between the individual point clusters and a deviation between normals associated with the individual point clusters;
wherein generating one or more tool paths is based on traversing the graph.

8. The system of claim 7 , wherein the control system is configured to generate one or more tool paths by:
determining a first point cluster and a second point cluster having a furthest separation in the graph;
generating a first tool path between the first point cluster and the second point cluster;
determining that the first tool path does not completely cover the indication of the defect;
in response to determining that the first tool path does not completely cover the indication of the defect, identifying two additional point clusters having a furthest separation amongst point cluster that can be connected by a second tool path that does not cross the first tool path; and
generating the second tool path between the two additional point clusters.

9. The system of claim 1 , wherein the control system is configured to generate one or more control signals by:
determining a plurality of possible tool positions for accessing the set of 3D coordinates;
determining a subset of viable tool positions for each point cluster based on collision information; and
selecting a particular tool position for each point cluster based on predefined motion criteria;
wherein the predefined motion criteria provides a first constraint for a first point cluster relative to one or more other point clusters.

10. The system of claim 9 , wherein the control system is configured to determine a plurality of tool positions for accessing each 3D coordinate of the set by:
defining a virtual camera with a field of view extending outward from the surface of the workpiece at the 3D coordinate; and
defining a plurality of lines that extend from the 3D coordinate within the field of view of the virtual camera, each line intersecting the surface of the workpiece at a tool angle.

11. The system of claim 10 , wherein the control system is configured to determine the subset of viable tool positions for each 3D coordinate based on collision information by:
determining for each line, whether access to the 3D coordinate at the tool angle of the line would generate a collision; and
selectively including each line in the subset of viable tool positions based on whether the line is associated with a collision.

12. The system of claim 1 , wherein the manipulation tool comprises a brush.

13. A system, comprising:
one or more robotic manipulators configured to support a manipulation tool for accessing a workpiece;
one or more image capture devices configured to generate two-dimensional images of

Assignees

• Gen Electric

Inventors

• Harel Stephane
• Bouchard Steeves
• Karigiannis John
• Lizotte Antoine
• Montplaisir Marcandre
• Saudrais Nicolas
• Cantin David

Classifications

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06V2201/06

  Recognition of objects for industrial automation · CPC title

• G01N21/9515

  Objects of complex shape, e.g. examined with use of a surface follower device (measuring contours and curvatures G01B11/24) · CPC title

• B25J9/1682 Primary

  Dual arm manipulator; Coordination of several manipulators · CPC title