Method, system, and device for removing smoke from laparoscope images based on conditional diffusion model

Patent metadata

Field	Value
Publication number	US-12376730-B1
Application number	US-202418985040-A
Country	US
Kind code	B1
Filing date	Dec 18, 2024
Priority date	Jan 29, 2024
Publication date	Aug 5, 2025
Grant date	Aug 5, 2025

Abstract

Disclosed are a method, system, and device for removing smoke from laparoscope images based on a conditional diffusion model. The method includes: segmenting a video of a laparoscopic surgery according to the number of frames to form a data set; performing smoke rendering on the obtained laparoscope smokeless images, and synthesizing paired smoky images to obtain a synthetic data set containing the smokeless images and the smoky images; inputting the smokeless images into the conditional diffusion model for forward noise addition, and continuously adding noise until the smokeless images are completely noised; inputting the smoky images into a smoke sensing module to obtain smoke concentration and position information, then training a neural network, and continuously performing reverse denoising on the completely noised images using the trained neural network until clear smokeless images are outputted; and optimizing a smoke removal model through a multi-loss function fusion strategy.

First claim

What is claimed is:

1. A method for removing smoke from laparoscope images based on a conditional diffusion model, comprising the following steps:
1) segmenting a video of a laparoscopic surgery according to the number of frames to form a data set in the form of pictures; performing smoke rendering on the obtained laparoscope smokeless images, and synthesizing paired smoky images to obtain a synthetic data set containing the smokeless images and the smoky images;
2) inputting the smokeless images into the conditional diffusion model for forward noise addition, and continuously adding noise until the smokeless images are completely noised to obtain a series of noisy images,
wherein a specific operation of the step 2) is as follows: inputting the smokeless images captured by a laparoscope into the conditional diffusion model for forward noise addition, wherein an added noise variance is β t ; the noise addition process follows a Markov chain, and a noise-added image is recorded as x t ; and performing noise addition on inputted smokeless images T times until the smokeless images are completely noised to obtain the series of noisy images {x 1 , x 2 , . . . , x t , . . . , x T }, wherein x t represents a noise image obtained by performing a t-th noise addition, and x T represents a noise image obtained by performing a T-th noise addition;
an equation for the forward noise addition process is as follows:

x
t

=

1
-

β
t

⁢

x

t
-
1

+

β
t

⁢
ε

,

ε
~

N
⁡
(

0
,
I

)

,

wherein ε represents noise, N represents a standard normal Gaussian distribution, I represents an identity matrix, and x t-1 represents a noise image obtained by performing a t−1-th noise addition;
3) inputting the smoky images into a smoke sensing module to obtain smoke concentration and position information, then training a neural network through the series of noisy images, and performing reverse denoising using the trained neural network; continuously performing reverse denoising on the completely noised images obtained in the step 2) until clear smokeless images are outputted,
wherein a specific operation of the step 3) is as follows: inputting the smoky images in the synthetic data set into the smoke sensing module to obtain smoke mask information and dark channel prior (DCP) information; taking the series of noisy images in the step 2) as labels for reverse training, wherein the reverse training process of each label is carried out through a U-Net network with the addition of a feature frequency compensation module FCB to jointly complete the training of the U-Net network; taking guidance images and the series of noisy images as inputs of the reverse training to train the U-Net network, the guidance images comprising the smokeless images and the synthetic smoky images in the synthetic data set; and finally, performing reverse denoising using the trained neural network, taking the smoke mask information and the DCP information as condition information to guide the reverse denoising process, and continuously performing denoising on the inputted completely noised images finally obtained in the step 2) until clear laparoscope smokeless images are obtained;
the smoke sensing module comprises a smoke mask segmentation module and a DCP module, for smoke segmentation and smoke concentration information extraction on the inputted smoky images, respectively;
minimum values in three channels of R, G, and B are taken to form a grayscale image, and then minimum value filtering is performed to obtain a dark channel in the DCP module; then, a network acquires smoke distribution and concentration information in the smoke images through the DCP, and the obtained smoke information is used as condition information to guide the reverse denoising in the diffusion model to generate related smokeless images;
related calculation equations are as follows:
M c ( F )=σ( MLP (AvgPool( F )+ MLP (MaxPool( F ))),

M s ( F )=σ( f 7×7 ([AvgPool( F );MaxPool( F )])),

F′=M c ⊙( M c ⊙F ),

wherein M c (F) represents channel attention, M s (F) represents spatial attention, F represents a depth feature, σ represents a sigmoid activation function, AvgPool represents average pooling, MaxPool represents maximum pooling, MLP represents an activation function, f 7×7 represents a convolution operation of a 7+7 filter, [⋅, ⋅] represents connection of feature maps, (represents element multiplication, and F′ represents a feature map;
the feature frequency compensation module FCB comprises a plurality of convolution filters, and bandwidth of these filters covers mid- and high-frequency components that are difficult to capture in the network; the calculation of the feature frequency compensation module FCB is as follows:
f k,σ =G k×k σ *f, k∈{ 3,5,7,9 . . . },

wherein G k×k σ represents a two-dimensional Gaussian kernel having a mean value of σ and a size of k, f represents an inputted feature value, and f k,σ represents a convolution output through the two-dimensional Gaussian kernel; four Gaussian kernels having sizes of 3, 5, 7, and 9 are selected for filtering, and a filter is obtained after making a difference between two Gaussian kernels, which is calculated and expressed by an equation as:
f k ′={f k ′,f 3 ′,f 5 ′,f 7 ′,f 9 ′}={f,f−f 3 ,f 3 −f 5 ,f 5 −f 7 ,f 7 −f 9 },

wherein f k ′ represents an output after filtering, f 9 ′ represents frequency information after being filtered by the Gaussian kernel having a size of 9, f 7 ′ represents frequency information after being filtered by the Gaussian kernel having a size of 7, f 5 ′ represents frequency information after being filtered by the Gaussian kernel having a size of 5, and f 3 ′ represents frequency information after being filtered by the Gaussian kernel having a size of 3;
filtering of different frequency bands is then realized by selecting different Gaussian kernels, and f is weighted and summarized as follows:

f
_

=<
W

,

f
k
′

>=

W
1

⁢

Assignees

• Univ Shandong

Inventors

• Huang Pu
• Li Dengwang
• Xue Jie
• Cheng Yao
• Jin Bin
• Niu Haitao
• Zhang Guangyong
• Zhai Xiangyu
• Li Hao
• Tian Baolong
• Nie Linchuan

Classifications

• G06T7/11

  Region-based segmentation · CPC title

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G06T2207/10068

  Endoscopic image · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title