Systems and methods of feature correspondence analysis

Patent metadata

Field	Value
Publication number	US-10937150-B2
Application number	US-201816022074-A
Country	US
Kind code	B2
Filing date	Jun 28, 2018
Priority date	Jun 28, 2018
Publication date	Mar 2, 2021
Grant date	Mar 2, 2021

Abstract

A method and system, the method including receiving semantic descriptions of features of an asset extracted from a first set of images; receiving a model of the asset, the model constructed based on a second set of a plurality images of the asset; receiving, based on an optical flow-based motion estimation, an indication of a motion for the features in the first set of images; determining a set of candidate regions of interest for the asset; determining a region of interest in the first set of images; iteratively determining a matching of features in the set of candidate regions of interest and the determined region of interest in the first set of images to generate a record of matches in features between two images in the first set of images; and displaying a visualization of the matches in features between two images in the first set of images.

First claim

What is claimed is:

1. A system, comprising:
a memory storing executable program instructions therein; and
a processor in communication with the memory, the processor operative to execute the program instructions to:
receive semantic descriptions of features of an industrial asset, wherein the features are extracted from a first set of images of the industrial asset;
generate a model of the industrial asset, wherein the model is constructed based on a second set of images of the industrial asset and describes a three-dimensional (3D) region of interest in the industrial asset in each image in the second set of images;
receive, based on an optical flow-based motion estimation, an indication of motion for the features in the first set of images of the industrial asset;
determine, based on the model and a third set of images of the industrial asset, a set of candidate regions of interest in the first set of images;
determine, based on the indication of motion for the features in the first set of images, a region of interest in the first set of images;
iteratively determine a matching of features in the set of candidate regions of interest in the first set of images and the region of interest in the first set of images to generate a record of matches in features between two images in the first set of images; and
display a visualization of the matches in features between the two images in the first set of images.

2. The system of claim 1 , wherein images included in the second set of images used to construct the model are captured from a plurality of different view-angles.

3. The system of claim 1 , wherein the semantic descriptions of the features of the industrial asset comprise multiple levels of description including a low level description that describes basic primitive features, a middle level description that describes objects as a group of features, and a high level description that describes the objects and affordances of the objects.

4. The system of claim 1 , wherein the semantic descriptions of the features of the industrial asset are determined by a deep learning neural network.

5. The system of claim 1 , wherein the third set of images of the industrial asset are obtained during a runtime execution of the system and include a stream of updated images of the industrial asset that are used to update the model.

6. The system of claim 1 , wherein the record of the matches in features between the two images in the first set of images includes two time-wise consecutive images.

7. The system of claim 1 , wherein the semantic descriptions of the features of the industrial asset comprise affordances of the features.

8. A computer-implemented method, comprising:
receiving semantic descriptions of features of an industrial asset, wherein the features are extracted from a first set of images of the industrial asset;
generating a model of the industrial asset, wherein the model is constructed based on a second set of images of the industrial asset and describes a three-dimensional (3D) region of interest in the industrial asset in each image in the second set of images;
receiving, based on an optical flow-based motion estimation, an indication of motion for the features in the first set of images of the industrial asset;
determining, based on the model and a third set of images of the industrial asset, a set of candidate regions of interest in the first set of images;
determining, based on the indication of motion for the features in the first set of images, a region of interest in the first set of images;
iteratively determining a matching of features in the set of candidate regions of interest in the first set of images and the region of interest in the first set of images to generate a record of matches in features between two images in the first set of images; and
displaying a visualization of the matches in features between the two images in the first set of images.

9. The computer-implemented method of claim 8 , wherein images included in the second set of images used to construct the model are captured from a plurality of different view-angles.

10. The computer-implemented method of claim 8 , wherein the semantic descriptions of the features of the industrial asset comprise multiple levels of description including a low level description that describes basic primitive features, a middle level description that describes objects as a group of features, and a high level description that describes the objects and affordances of the objects.

11. The computer-implemented method of claim 8 , wherein the semantic descriptions of the features of the industrial asset are determined by a deep learning neural network.

12. The computer-implemented method of claim 8 , wherein the third set of images of the industrial asset are obtained during a runtime execution and include a stream of updated images of the industrial asset that are used to update the model.

13. The computer-implemented method of claim 8 , wherein the record of the matches in features between the two images in the first set of images includes two time-wise consecutive images.

14. A non-transitory computer readable medium having executable instructions stored therein, the non-transitory computer readable medium comprising:
instructions to receive semantic descriptions of features of an industrial asset, wherein the features are extracted from a first set of images of the industrial asset;
instructions to generate a model of the industrial asset, wherein the model is constructed based on a second set of images of the industrial asset and describes a three-dimensional (3D) region of interest in the industrial asset in each image in the second set of images;
instructions to receive, based on an optical flow-based motion estimation, an indication of motion for the features in the first set of images of the industrial asset;
instructions to determine, based on the model and a third set of images of the industrial asset, a set of candidate regions of interest in the first set of images;
instructions to determine, based on the indication of motion for the features in the first set of images, a region of interest in the first set of images;
instructions to iteratively determine a matching of features in the set of candidate regions of interest in the first set of images and the region of interest in the first set of images to generate a record of matches in features between two images in the first set of images; and
instructions to display a visualization of the matches in features between the two images in the first set of images.

15. The non-transitory computer readable medium of claim 14 , wherein images included in the second set of images used to construct the model are captured from a plurality of different view-angles.

16. The non-transitory computer readable medium of claim 14 , wherein the semantic descriptions of the features of the industrial comprise multiple levels of description including at least a low level description that describes basic primitive features, a middle level description that describes objects as a group of features, and a high level description that describes the objects and affordances of the objects.

17. The non-transitory computer readable medium of claim 14 , wherein the semantic descriptions of the features of the industrial asset are determined by a deep learning neural network.

18. The non-transitory computer readable medium of claim 14 , wherein the third set of images of the industrial asset are obtained during a runtime execution and include a strea

Assignees

• Gen Electric

Inventors

• Tan Huan
• Jain Arpit
• Cheon Gyeong Woo
• Baloch Ghulam Ali
• Tu Jilin
• Ge Weina
• Zhang Li

Classifications

• G06V10/759

  Region-based matching · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06T7/251 Primary

  involving models · CPC title

• G06V10/757

  Matching configurations of points or features · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title