Systems and methods of 3D scene segmentation and matching for robotic operations

Patent metadata

Field	Value
Publication number	US-10970586-B2
Application number	US-201816022160-A
Country	US
Kind code	B2
Filing date	Jun 28, 2018
Priority date	Jun 28, 2018
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

A method and system, the method including receive image data representations of a set of images of a physical asset; receive a data model of at least one asset, the data model of each of the at least one assets including a semantic description of the respective modeled asset and at least one operation associated with the respective modeled asset; determine a match between the received image data and the data model of one of the at least one assets based on a correspondence therebetween; generate, for the data model determined to be a match with the received image data, an operation plan based on the at least one operation included in matched data model; execute, in response to the generation of the operation plan, the generated operation plan by the physical asset.

First claim

What is claimed is:

1. A system, comprising:
a memory storing executable program instructions therein; and
a processor in communication with the memory, the processor operative to execute the program instructions to:
receive image data comprising a set of images of a physical asset;
receive data models of a plurality of assets, wherein each of the data models includes a semantic description of a respective asset, and wherein the semantic description includes at least one operation associated with the respective asset;
determine a match between the physical asset represented in the image data and a corresponding data model of the data models;
generate, upon determining the match between the physical asset and the corresponding data model, an operation plan based on a corresponding operation included in the corresponding data model;
execute the operation plan via a robot configured to interact with the physical asset;
quantify a result of execution of the operation plan by the robot;
update the corresponding operation included in the corresponding data model based on the result to generate an updated operation associated with the corresponding data model and different than the corresponding operation; and
store the updated operation in an operation library.

2. The system of claim 1 , wherein determining the match between the physical asset represented in the image data and the corresponding data model comprises:
extracting features from the image data; and
comparing the features extracted from the image data with features of the corresponding data model.

3. The system of claim 1 , wherein the at least one operation comprises a plurality of categories of operations.

4. The system of claim 3 , wherein the plurality of categories of operations comprises a manipulation operation defining how the robot operates and moves a manipulator, a navigation operation defining how the robot moves relative to another entity, and a perception operation defining how the robot obtains and processes sensory information.

5. The system of claim 1 , wherein generating the operation plan based on the corresponding operation comprises:
extracting search terms from the semantic description associated with the corresponding data model; and
querying a database using the search terms for identification of operations to be executed by the robot configured to perform a task on the physical asset.

6. A computer-implemented method comprising:
receiving image data comprising a set of images of a physical asset;
receiving data models of a plurality of assets, wherein each of the data models includes a semantic description of a respective asset, and wherein the semantic description includes at least one operation associated with the respective asset;
determining a match between the physical asset represented in the image data and a corresponding data model of the data models;
generating, upon determining the match between the physical asset and the corresponding data model, an operation plan based on a corresponding operation included in the corresponding data model;
executing the operation plan via a robot configured to interact with the physical asset;
quantifying a result of execution of the operation plan by the robot;
updating the corresponding operation included in the corresponding data model based on the result to generate an updated operation associated with the corresponding data model and different than the corresponding operation; and
storing the updated operation in an operation library.

7. The computer-implemented method of claim 6 , wherein determining the match between the physical asset represented in the image data and the corresponding data model comprises:
extracting features from the image data; and
comparing the features extracted from the image data with features of the corresponding data model.

8. The computer-implemented method of claim 6 , wherein the at least one operation comprises a plurality of categories of operations.

9. The computer-implemented method of claim 8 , wherein the plurality of categories of operations comprises a manipulation operation defining how the robot operates an actuator, a navigation operation defining how the robot can move relative to another entity, and a perception operation defining how the robot obtains and processes sensory information.

10. The computer-implemented method of claim 6 , wherein generating the operation plan based on the corresponding operation comprises:
extracting search terms from the semantic description associated with the corresponding data model; and
querying a database using the search terms for identification of operations to be executed by the robot configured to perform a task on the physical asset.

11. A non-transitory computer readable medium having executable instructions stored therein, the non-transitory computer readable medium comprising:
instructions to receive image data comprising a set of images of a physical asset;
instructions to receive data models of a plurality of assets, wherein each of the data models includes a semantic description of a respective asset, and wherein the semantic description includes at least one operation associated with the respective asset;
instructions to determine a match between the physical asset represented in the image data and a corresponding data model of the data models;
instructions to generate, upon determining the match between the physical asset and the corresponding data model, an operation plan based on a corresponding operation included in the corresponding data model;
instructions to execute the operation plan via a robot configured to interact with the physical asset;
instructions to quantify a result of execution of the operation plan by the robot;
instructions to update the corresponding operation included in the corresponding data model based on the result to generate an updated operation associated with the corresponding data model and different than the corresponding operation; and
instructions to store the updated operation in an operation library.

12. The non-transitory computer readable medium of claim 11 , wherein determining the match between the physical asset represented in the image data and the corresponding data model comprises:
extracting features from the image data; and
comparing the features extracted from the image data with features of the corresponding data model.

13. The non-transitory computer readable medium of claim 11 , wherein the at least one operation comprises plurality of categories of operations.

14. The non-transitory computer readable medium of claim 13 , wherein the plurality of categories of operations comprises a manipulation operation defining how the robot operates an actuator, a navigation operation defining how the robot can move relative to another entity, and a perception operation defining how the robot obtains and processes sensory information.

15. The non-transitory computer readable medium of claim 11 , wherein generating the operation plan based on the corresponding operation comprises:
extracting search terms from the semantic description associated with the corresponding data model; and
querying a database using the search terms for identification of operations to be executed by the physical asset.

Assignees

• Gen Electric

Inventors

• Tan Huan
• Cheon Gyeong Woo
• Tu Jilin
• Zhang Li

Classifications

• G06F16/583 Primary

  using metadata automatically derived from the content · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V20/653

  by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces · CPC title