Symbol recognition from raster images of PandIDs using a single instance per symbol class

Patent metadata

Field	Value
Publication number	US-12039641-B2
Application number	US-202217722527-A
Country	US
Kind code	B2
Filing date	Apr 18, 2022
Priority date	Jun 25, 2021
Publication date	Jul 16, 2024
Grant date	Jul 16, 2024

Abstract

Traditional systems that enable extracting information from Piping and Instrumentation Diagrams (P&IDs) lack accuracy due to existing noise in the images or require a significant volume of annotated symbols for training if deep learning models that provide good accuracy are utilized. Conventional few-shot/one-shot learning approaches require a significant number of training tasks for meta-training prior. The present disclosure provides a method and system that utilizes the one-shot learning approach that enables symbol recognition using a single instance per symbol class which is represented as a graph with points (pixels) sampled along the boundaries of different symbols present in the P&ID and subsequently, utilizes a Graph Convolutional Neural Network (GCNN) or a GCNN appended to a Convolutional Neural Network (CNN) for symbol classification. Accordingly, given a clean symbol image for each symbol class, all instances of the symbol class may be recognized from noisy and crowded P&IDs.

First claim

What is claimed is:

1. A processor implemented method comprising the steps of:
receiving as an input, via one or more hardware processors, at least a portion of a Piping and Instrumentation Diagram (P&ID) as a raster graphics (RG) image;
converting, via the one or more hardware processors, the raster graphics (RG) image to a vector graphics (VG) image, wherein contours of a plurality of entities comprised in the VG image are represented by sequential Bezier curves, and wherein the plurality of entities includes texts, symbols and pipelines; and
segmenting, via the one or more hardware processors, the symbols from the VG image, wherein the sequential Bezier curves along the contours of the plurality of entities represent one or more paths comprising one or more loops such that ends thereof are connected, wherein the step of segmenting the symbols comprises:
sampling a set of points sequentially along the one or more paths at predetermined fixed distance intervals, each point in the sampled set of points being characterized by a slope;
eliminating the one or more paths corresponding to the texts based on (i) a length associated with the one or more paths computed using the sampled set of points and (ii) a computed segregation threshold, wherein the computed segregation threshold is a knee point of a curve representing the number of sampled points in each of the one or more paths (Y-axis) versus a cardinal value of the one or more paths (X-axis); and
eliminating the one or more paths corresponding to the pipelines using a sliding window method.

2. The processor implemented method of claim 1 , wherein the step of sampling the set of points is preceded by preprocessing of the VG image for at least one of rotation and scaling variations.

3. The processor implemented method of claim 1 , wherein the step of eliminating the one or more paths corresponding to the texts comprises:
computing the length associated with the one or more paths for each entity from the plurality of entities based on the number of sampled points in each of the one or more paths, the number of sampled points being proportional to the length of a corresponding path; and
identifying the one or more paths having a computed length lesser than the computed length associated with the knee point as the one or more paths corresponding to the texts for elimination.

4. The processor implemented method of claim 1 , wherein the step of eliminating pipelines using a sliding window method comprises:
querying each point p i from the sampled set of points, for the presence of other points in a sliding window, wherein the sliding window is of length 1 and height t in an orthogonal direction to the slope of the point p i ; and
classifying the queried point p i as a component of the pipelines for elimination, if a standard deviation of points in the sliding window is within a predetermined threshold.

5. The processor implemented method of claim 3 , wherein the height t is determined by traversing the one or more paths and finding a maximum distance from the sampled set of points along the orthogonal direction to the queried point.

6. The processor implemented method of claim 4 , wherein the slope of the point p i with reference to two neighboring points and p i−1 and p i+1 is represented as:
slope i =Avg( a tan 2( p i−1 ,p i ), a tan 2( p i ,p i+1 ), a tan 2( p i−1 ,p i+1 )); and
the height t is represented as:
t=MAX(∈,β MAX(∀p iN dist(p i ,p iN ))), wherein p iN are points along the paths orthogonal to a query point p i .

7. The processor implemented method of claim 1 , further comprising classifying, via the one or more hardware processors, the segmented symbols using (i) a pre-trained Graph Convolutional Neural Network (GCNN), wherein the segmented symbols are represented by an associated point cloud comprising the sampled set of points or (ii) the pre-trained GCNN appended to a Convolutional Neural Network (CNN).

8. The processor implemented method of claim 6 , wherein the step of classifying the segmented symbols is preceded by training a GCNN, the step of training comprising:
receiving a set of symbol classes comprising a single instance per symbol class in the form of the VG image;
augmenting the single instance per symbol class by applying affine transformation to each sub-part of the received VG image with rotation ranging from an angle −20° to 20°, scaling parameter ranging from 0.9 to 1.1 and shear parameter ranging from −0.05 to 0.05 to obtain an augmented set of instances for each symbol class;
obtaining a plurality of features for each point in (i) the received VG image and (ii) the augmented set of instances for each symbol class, wherein the plurality of features includes: (i) two features corresponding to a coordinate information associated with each point and (ii) seven features corresponding to seven Hu moments for each point; and
training the GCNN, by inputting a graph generated for every instance of a symbol class, using each point in (i) the received VG image and (ii) the augmented set of instances for each symbol class, each point having the obtained plurality of features, to obtain the pre-trained GCNN.

9. The processor implemented method of claim 7 , wherein the step of classifying the segmented symbols is preceded by training the CNN, using an image created from each point in (i) the received VG image and (ii) the augmented set of instances for each symbol class for obtaining a global embedding of the point cloud thereof.

10. A system comprising:
a memory storing instructions;
one or more communication interfaces; and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
receive as an input, at least a portion of a Piping and Instrumentation Diagram (P&ID) as a raster graphics (RG) image;
convert the raster graphics (RG) image to a vector graphics (VG) image, wherein contours of a plurality of entities comprised in the VG image are represented by sequential Bezier curves, and wherein the plurality of entities includes texts, symbols and pipelines; and
segment the symbols from the VG image, wherein the sequential Bezier curves along the contours of the plurality of entities represent one or more paths comprising one or more loops such that ends thereof are connected, wherein the symbols are segmented by:
sampling a set of points sequentially along the one or more paths at predetermined fixed distance intervals, each point in the sampled set of points being characterized by a slope;
eliminating the one or more paths corresponding to the texts based on (i) a length associated with the one or more paths computed using the sampled set of points and (ii) a computed segregation threshold being a knee point of a curve representing the number of sampled points in each of the one or more paths (Y-axis) versus a cardinal value of the one or more paths (X-axis), wherein the computed segregation threshold is a knee point of a curve representing the number of sampled points in each of the one or more paths (Y-axis) versus a cardinal value of the one or more paths (X-axis); and
eliminating the one or more paths corresponding to the pipelines using a sliding window method.

11. The system of claim 10 , wherein the one or more processors are configured by the instructions to preprocess the VG image for at least one of rotation and scaling variations prior to sampling the set of points.

12. The system of claim 10 , wherein the one or more processors are configured by the instructions to eliminate the one or more paths corresponding to the

Assignees

• Tata Consultancy Services Ltd
• Tata Consultancy Ltd Services

Inventors

• Paliwal Shubham Singh
• VIG Lovekesh
• Sharma Monika

Classifications

• G06T11/23 Primary

  using straight lines or curves · CPC title

• G06T3/147

  using affine transformations · CPC title

• G06T11/40

  Filling planar surfaces by adding surface attributes, e.g. adding colours or textures · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title