Portable edge AI-assisted diagnosis and quality control system for gastrointestinal endoscopy

Patent metadata

Field	Value
Publication number	US-12562279-B2
Application number	US-202418591111-A
Country	US
Kind code	B2
Filing date	Feb 29, 2024
Priority date	Feb 29, 2024
Publication date	Feb 24, 2026
Grant date	Feb 24, 2026

Abstract

In a decision-support system for gastrointestinal (GI) endoscopy, convolutional neural networks (CNNs) are set up to perform decision-support tasks according to endoscopic images. Each learnable kernel used in the CNNs is advantageously modeled as a linear combination of a set of fixed kernels for simplifying kernel learning, giving a lightweight kernel model to advantageously reduce required computation resources. Further computation-resource reduction can be made by CNN model compression via knowledge distillation and by using multi-task CNNs. It enables the decision-support system to be realized as an edge computing system near a site of performing endoscopic examinations. The system can be automatically configured for esophagogastroduodenoscopy (EGD) or colonoscopy. In the system, lesion-detection results and quality-control results can be seamlessly integrated to provide value-added results, which are more valuable to the endoscopist than separately considering the lesion-detection results and quality-control results.

First claim

What is claimed is:

1 . A decision-support system for gastrointestinal (GI) endoscopy, the system comprising one or more computers configured to execute a computing process for processing a sequence of endoscopic images acquired in a GI endoscopic examination to at least perform a plurality of selected tasks dynamically selected from a plurality of decision-support tasks, the computing process comprising:
setting up a plurality of convolutional neural networks (CNNs) for performing the plurality of decision-support tasks according to the sequence of endoscopic images, an individual CNN being modeled with one or more learnable kernels, wherein each learnable kernel used in the plurality of CNNs is modeled as a linear combination of a set of fixed kernels with the set of fixed kernels being invariant over the plurality of CNNs so as to simplify kernel learning in comparison to training a conventional CNN model with unrestricted one or more kernels to thereby reduce a computation-resource requirement of the one or more computers and hence enable the decision-support system to be realized as an edge computing system near a site of performing the GI endoscopic examination; executing a subprocess, wherein the subprocess comprises performing the plurality of selected tasks by processing the sequence of endoscopic images with each CNN in a plurality of selected CNNs, the plurality of selected CNNs being identified from the plurality of CNNs and being used for performing the plurality of selected tasks; and repeating the subprocess until an event indicative of exiting from looping the subprocess occurs.

2 . The decision-support system of claim 1 , wherein each kernel in the set of fixed kernels is a 3×3 matrix.

3 . The decision-support system of claim 1 , wherein the computing process further comprises compressing an individual selected CNN in the plurality of selected CNNs by knowledge distillation to reduce a model complexity of the individual selected CNN such that the individual selected CNN as compressed is used to process the sequence of endoscopic images to perform one or more corresponding decision-support tasks associated with the individual selected CNN to thereby further reduce the computation-resource requirement of the one or more computers.

4 . The decision-support system of claim 1 , wherein:
the plurality of CNNs includes one or more multi-task CNNs, an individual multi-task CNN being used for performing plural corresponding decision-support tasks in the plurality of decision-support tasks; and in the setting up of the plurality of CNNs, at least one multi-task CNN is formed with one or more layers shared by the plural corresponding decision-support tasks to thereby further reduce the computation-resource requirement of the one or more computers.

5 . The decision-support system of claim 1 , wherein:
the plurality of CNNs includes one or more multi-task CNNs, an individual multi-task CNN being used for performing plural corresponding decision-support tasks in the plurality of decision-support tasks; and in the setting up of the plurality of CNNs, at least one multi-task CNN is formed with a serial cascade of multi-task attention fusion networks to thereby further reduce the computation-resource requirement of the one or more computers.

6 . The decision-support system of claim 1 , wherein:
the plurality of decision-support tasks is partitioned into a first plurality of tasks performed in esophagogastroduodenoscopy (EGD), a second plurality of tasks performed in colonoscopy, and a third plurality of tasks performed in both EGD and colonoscopy; the third plurality of tasks includes the task of determining an imaging location in an upper or lower GI tract so as to determine whether EGD or colonoscopy is carried out; the computing process further comprises initializing the plurality of selected tasks as the third plurality of tasks before an initial execution of the subprocess; and the subprocess further comprises:
if the imaging location is determined to be in the upper GI tract during performing the plurality of selected tasks, then updating the plurality of selected tasks by including the first plurality of tasks and removing the second plurality of tasks; and
if the imaging location is determined to be in the lower GI tract during performing the plurality of selected tasks, then updating the plurality of selected tasks by including the second plurality of tasks and removing the first plurality of tasks, whereby the updating of the plurality of selected tasks from time to time provides automatic configuration of the decision-support system for EGD and colonoscopy.

7 . The decision-support system of claim 1 , wherein:
the plurality of decision-support tasks includes a plurality of preparation tasks, a plurality of quality-control tasks and a plurality of lesion-detection tasks; the plurality of preparation tasks includes tasks of: classifying an imaging location as an in vivo location or an in vitro one; detecting a region of interest (ROI) on an image captured at the imaging location; assessing an image quality achieved at the imaging location; and determining the imaging location in an upper or lower GI tract so as to determine whether esophagogastroduodenoscopy (EGD) or colonoscopy is carried out; the plurality of quality-control tasks includes tasks of: assessing a level of cleanliness at the imaging location; classifying a stomach site in EGD; classifying an anatomical landmark at the imaging location in colonoscopy; and estimating a withdrawal speed in colonoscopy; and the plurality of lesion-detection tasks includes tasks of: detecting a lesion in EGD; identifying a cancer in EGD; detecting Helicobacter pylori (HP) infection in EGD; detecting polyp/adenoma in colonoscopy; and identifying a cancer in colonoscopy.

8 . The decision-support system of claim 7 , wherein the anatomical landmark is classified as a terminal ileum, a cecum, an ascending colon, a traverse colon, a descending colon, a sigmoid colon, a rectum, or an anus.

9 . The decision-support system of claim 7 , wherein the computing process further comprises performing one or more reporting tasks selected from tasks of: reporting the assessed level of cleanliness; reporting a HP degree in EGD; reporting a level of cancer risk; reporting key images of lesion/polyp as extracted from the sequence of endoscopic images; and reporting key images of stomach sites/anatomical landmarks as extracted from the sequence of endoscopic images.

10 . The decision-support system of claim 1 , wherein:
the plurality of decision-support tasks includes a plurality of lesion-detection tasks; the plurality of lesion-detection tasks includes a task of detecting a lesion in EGD; and in the setting up of the plurality of CNNs, a corresponding CNN associated with the task of detecting the lesion in EGD is a visual attention network (VAN).

11 . The decision-support system of claim 1 , wherein:
the plurality of decision-support tasks includes a plurality of lesion-detection tasks; the plurality of lesion-detection tasks includes a task of detecting polyp/adenoma in colonoscopy; and in the setting up of the plurality of CNNs, a corresponding CNN associated with the task of detecting polyp/adenoma in colonoscopy is a visual transformer (ViT) utilizing coordinate attention (CA).

12 . The decision-support system of claim 1 , wherein:
the plurality of decision-support tasks includes a plurality of quality-control tasks; the plurality of quality-control tasks includes a task of classifying an anatomical landmark at an imaging location in colonoscopy; and in the setting up of the plurality of

Assignees

• Hong Kong Applied Science & Tech Research Inst Co Ltd

Inventors

• He Xuejian
• Wang Lu
• Leung Ping Shun
• Liu Shangping
• Chow Chi Hin Samuel
• Liu Xindong
• Bangalore Nagaraj Manu

Classifications

• G06V2201/031

  of internal organs · CPC title

• G06T2207/30096

  Tumor; Lesion · CPC title

• G06T2207/30032

  Colon polyp · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title