Intelligent recognition method for time sequence image of concrete dam defect

Patent metadata

Field	Value
Publication number	US-12536648-B2
Application number	US-202318322605-A
Country	US
Kind code	B2
Filing date	May 24, 2023
Priority date	May 11, 2022
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

Disclosed is an intelligent recognition method for a time sequence image of a concrete dam defect. The method includes: extracting a feature sequence of the time sequence image containing the concrete dam defect; matching a located defect with a real defect by using an objective function; adding a loss term based on a tight sensing intersection-over-union to a loss function of a model so as to pay attention to integrity of a defect sequence and improve accuracy; and extracting a defect feature and recognizing a defect type after completing defect location. According to the present disclosure, the time sequence image of the concrete dam defect is detected effectively, so that a defect in a long image sequence can be located and the defect type can be recognized accurately.

First claim

What is claimed is:

1 . An intelligent recognition method for a time sequence image of a concrete dam defect, comprising:
(1) training a defect location model for a time sequence image characteristic containing a dam defect; (2) extracting, by the defect location model, a time sequence feature by using a two-stream network and a Transformer network, wherein the extracting further comprises: a. extracting the image feature by using the two-stream network; b. adding a time-dimensional self-attention mechanism to an image frame through the Transformer network; and c. obtaining a global feature relation, so as to identify a located defect; (3) matching the located defect with a real defect using an objective function based on a distance intersection-over-union; wherein the matching further comprises: a. computing a defect position relation so as to accelerate a model convergence rate; b. adding a loss term based on a tight sensing intersection-over-union to a loss function; and c. paying attention, using the self-attention mechanism, to integrity of a defect sequence so as to improve accuracy of defect location; (4) locating the defect sequence, wherein the locating further comprises: a. identifying a defect image frame from the defect sequence using a convolutional network based on a two dimensional (2D) time sequence difference; and b. recognizing a defect type by extracting visual and displacement information from the defect image frame.

2 . The intelligent recognition method for a time sequence image of a concrete dam defect according to claim 1 , wherein the extracting a time sequence feature by using a two-stream network and a Transformer network further comprises:
(1.1) inputting an original time sequence image, denoted by X={x n } n=1 l , wherein the sequence contains an image frame, and x n represents an n th frame of the sequence X; (1.2) converting the original time sequence image into S n =(x t n , o t n ) as an input of the two-stream network, wherein x t n represents a red green blue (RGB) image of a t n frame of the original sequence image X; processing through a spatial stream convolutional network, wherein o t n =d t n (u, v) represents an optical flow stacked by RGB images of the t n frame and a t n +1 frame; and processing through a temporal stream convolutional network, wherein d t n x (u, v) and d t n y (u, v) represent horizontal and vertical displacement vectors of the t n +1 frame at a point (u, v), respectively, and are regarded as two input channels of a convolutional neural network; (1.3) denoting a feature sequence of a time sequence image extracted by the two-stream network by

F
=

{

f

t
n

}

n
=
1

l
s

,

and constituting a boundary assessment network by three convolutional layers; computing probabilities

P
s

=

{

p

t
n

s

}

n
=
1

l
s

⁢

and
⁢

P
E

=

{

p

t
n

e

}

n
=
1

l
s

of each frame as a start frame and an end frame of the defect sequence; and multiplying and combining an input feature of the time sequence image and the predicted probabilities, corresponding to each time sequence position, of start and end of the defect to obtain a feature sequence;
(1.4) adding a position code so as to mark the time sequence position for each frame, and computing a global self-attention weight for each frame by using the Transformer network to obtain a feature sequence of a defect image containing the attention weight; and
(1.5) predicting an image sequence of the feature sequence of the defect image containing the attention weight by using multi-layer perceptron, and outputting positions of the start frame and the end frame.

3 . The intelligent recognition method for a time sequence image of a concrete dam defect according to claim 1 , wherein the matching further comprises:
(2.1) matching, in the training process of the model, the located defect {circumflex over (φ)} j with the real defect φ j pairwise, and then computing an interval error as a loss value so as to optimize the model; and computing, in a matching process, an optimal match by maximizing the objective function, wherein the objective function is as follows:

π
=

arg
⁢

max
⁢

∑

n
=
1

N

(

DIoU
⁡
(

φ
n

,

φ
^

n

)

-

l
1

(

φ
n

,

Assignees

• Univ Hohai
• Huaneng Lancang River Hydropower Inc
• Huaneng Lanceng River Hydropower Inc

Inventors

• Ma Hongqi
• Zhou Hua
• Mao Yingchi
• Chi Fudong
• Zhou Xiaofeng
• Cao Xuexing
• Qi Rongzhi
• Chen Hao
• Tan Bin
• Nie Bingbing

Classifications

• G06V2201/07

  Target detection · CPC title

• G06T2207/30132

  Masonry; Concrete · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10024

  Color image · CPC title