Transmission line defect identification method based on saliency map and semantic-embedded feature pyramid

Patent metadata

Field	Value
Publication number	US-12299974-B2
Application number	US-202218081368-A
Country	US
Kind code	B2
Filing date	Dec 14, 2022
Priority date	May 9, 2022
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

The present disclosure provides a transmission line defect identification method based on a saliency map and a semantic-embedded feature pyramid, including the following steps: step 1: cleaning and classifying a dataset; step 2: generating a super-resolution image for a small target of a transmission line by using an Electric Line-Enhanced Super-Resolution Generative Adversarial Network (EL-ESRGAN) model; step 3: performing image saliency detection on the dataset by constructing a U 2 -Net; step 4: performing data augmentation on the dataset by using GridMask and random cutout algorithms based on a saliency map, and generating a classified dataset; and step 5: performing image classification on a normal set and a defect set by using a ResNet34 classification algorithm and a deep semantic embedding (DSE)-based feature pyramid classification network.

First claim

What is claimed is:

1. A transmission line defect identification method based on a saliency map and a semantic-embedded feature pyramid, the method comprising the following steps:
1) taking a target image of a transmission line as a dataset, labeling, based on whether the transmission line has a defect, the dataset as a normal set or a defect set, and classifying the dataset as a small target set or a non-small target set based on a size of the target image and a given threshold;
2) performing image super-resolution expansion on the small target set by using an Electric Line-Enhanced Super-Resolution Generative Adversarial Network (EL-ESRGAN) algorithm, combining the non-small target set and the small target set obtained after image super-resolution expansion, compressing a combined set based on a size of the small target set, and dividing the combined set into a training set and a test set;
3) generating the saliency map of an image in the training set by using a nested saliency detection network (U 2 -Net), ensuring integrity of a key region of a detection target by using a morphological expansion algorithm, generating a cutout region randomly for a part whose saliency score is less than a threshold, and padding a pixel randomly to form a data-augmented image set;
4) inputting a data-augmented image and its label into a deep semantic embedding (DSE)-based feature pyramid classification network to perform training to obtain a trained classifier; and
5) obtaining image data of an inspected target of the transmission line in real time, and taking the image data as an input of the trained classifier to output an identification result,
wherein performing the image super-resolution expansion on the small target further comprises:
defining loss functions of a generator and a discriminator of an EL-ESRGAN model, wherein formulas of the loss functions are as follows:
L G Ra =−E x r [log(1− D Ra ( x r ,x f ))]− E x f [log( D Ra ( x f ,x r ))]

L D Ra =−E x r [log( D Ra ( x r ,x f ))]− E x f [log(1− D Ra ( x f ,x r ))]

wherein L G Ra represents a GAN loss function of the generator, L D Ra represents a GAN loss function of the discriminator, D Ra (x r ,x f ) represents a probability that an authenticated image is more real than a false image, D Ra (x f ,x r ) represents a probability that the authenticated image is falser than a real image, E x f [ ] represents an averaging operation performed on all false data in a processing batch, x i represents a low-resolution image input into a GAN, x f represents an authenticated image that is generated by the GAN and determined to be false, and x r represents an authenticated image that is generated by the GAN and determined to be real;
training the generator of the EL-ESRGAN model by using the non-small target set of the transmission line to obtain a second-order degradation model, and using an L1 loss function, a perceptual loss function, and the GAN loss functions represented by L G Ra and L D Ra together to construct an overall loss function of the EL-ESRGAN, and performing training to obtain the EL-ESRGAN model; and
performing image super-resolution augmentation on the small target set of the transmission line by using the EL-ESRGAN model.

2. The according to claim 1 , further comprising:
building a residual U-block (RSU) network based on a residual block network structure;
building, by stacking the RSU network, the U 2 -Net composed of 11 stages;
generating a saliency score of the target image of the transmission line by using the U 2 -Net, ensuring integrity of the key region of the detection target by using the morphological expansion algorithm, and generating an image mask region; and
randomly selecting GridMask and random cutout algorithms to perform a cutout operation randomly in the image mask region, and padding the pixel randomly.

3. The method according to claim 1 , further comprising a DSE-based feature pyramid classification network comprising:
a residual network (ResNet) feature extraction module, wherein an input of the ResNet feature extraction module is the target image of the transmission line, and an output of the ResNet feature extraction module is features of different scales of the image;
an enhanced feature pyramid network (EFPN) module, wherein an input of the EFPN module is the features of the different scales that are generated by the ResNet feature extraction module, and an output of the EFPN module is a feature obtained by fusing features of adjacent scales;
a DSE module, where an input of the DSE module is the fused feature generated by the EFPN module, and an output of the DSE module is a low-resolution feature with rich semantic information and a high-resolution feature with rich position information;
a deep feature fusion (DFF) module, wherein an input of the DFF module is the low-resolution feature and the high-resolution feature generated by the DSE module, and an output of the DFF module is a feature vector obtained by performing convolution and pooling operations on the high-resolution feature and the low-resolution feature; and
an image object classification network (OC), wherein an input of the OC is the feature vector generated by the DFF module for the high-resolution feature and the low-resolution feature, and an output of the OC is a classification result indicating whether the inspected target of the transmission line is faulty.

4. The according to claim 1 , wherein the dataset is an insulator self-explosion dataset of the transmission line.

Assignees

• Univ Zhejiang

Inventors

• Yang Qiang
• Su Chao
• Cao Yuan
• Jiang Di
• Xu Hao
• Qiu Kaidi

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T5/20

  using local operators · CPC title

• G06T3/4053

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title