Real-time measurement of visible surface area from colonoscopy video

Patent metadata

Field	Value
Publication number	US-12064083-B2
Application number	US-202017600091-A
Country	US
Kind code	B2
Filing date	Apr 2, 2020
Priority date	Apr 5, 2019
Publication date	Aug 20, 2024
Grant date	Aug 20, 2024

Abstract

The present invention comprises a system and a computer-implemented method designed for measuring the visible colon surface area that has been seen from a colonoscope, and for displaying the results of the measurement. The method estimates the orientation of the camera of the colonoscope, estimates the travel distance of the scope camera from the axial vector lengths coming from or headed towards the focus of expansion point, and visualizing the percentage of the visible surface of the entire colon.

First claim

We claim:

1. A computer-implemented method comprising:
dividing a colon into axial sections of a given length;
dividing each axial section into a plurality of cross-sectional portions;
for each frame of video extracted from a camera as the camera is moved through each axial section:
determining to which cross-sectional portion of the axial section the camera is oriented in the video frame;
incrementing a counter for the cross-sectional portion to which the camera is oriented;
comparing the counter for each cross-sectional portion to an optimal value; and
visualizing the variance of the counter from the optimal value for each cross-sectional portion.

2. The method of claim 1 wherein the cross-sectional portions are quadrants.

3. The method of claim 1 wherein the cross-sectional portion of the axial section to which the camera is oriented is determined by a shadow area in the extracted video frame.

4. The method of claim 3 wherein the cross-sectional portion to which the camera is oriented is the cross-sectional opposite the cross-sectional portions containing the shadow area.

5. The method of claim 3 wherein the shape of the shadow area in the extracted frame is filtered with area size, compactness, and closing operation.

6. The method of claim 1 , further comprising:
determining that the end of an axial section has been reached by estimating the distance that the camera has moved through the axial section and comparing it with the given length of the axial section.

7. The method of claim 6 wherein estimating the distance that the camera has moved through the axial section between frames comprises:
calculating an optical flow comprising a vector pattern caused by relative motion between the camera and an interior surface of the colon as determined from successive extracted video frames;
projecting the vector pattern to an epi-polar plane; and
calculating an axial vector based on the projected vector pattern with respect to a focus of expansion.

8. The method of claim 7 wherein the axial travel direction of the camera is determined from the direction of the optical flow vectors with respect to the focus of expansion.

9. The method of claim 7 wherein the speed of the camera as it moves through the axial section is given by the magnitude of the axial vector.

10. The method of claim 1 wherein the visualization comprises, for each cross-sectional portion in each axial section, a cell comprising a heat map indicating the variance of the counter for each cross-sectional portion.

11. The method of claim 1 wherein the cells for each cross-sectional portion are arranged in a matrix of color rectangles wherein the rows of the matrix indicate the cross-sectional portions and the columns of the matrix indicate the axial sections.

12. The method of claim 11 wherein additional rows in the matrix include information regarding other measurements for each axial section.

13. The method of claim 12 wherein the other measurements include a clarity score, a colonoscopy preparation score, and a colon distention score.

14. An apparatus comprising:
a camera mounted on an endoscope;
a camera controller for controlling the orientation of the camera;
a scope computer connected to the endoscope and the camera controller for collecting video captured by the camera and for relaying control signals from the camera controller to the endoscope; and
a video analytic computer, coupled to the scope computer and receiving video captured by the camera from the scope computer, the video analytic computer executing software for performing the functions of:
dividing a colon into axial sections of a given length;
dividing each axial section into a plurality of cross-sectional portions;
for each axial section:
for each frame extracted from the video as the endoscope is moved through the axial section:
determining to which cross-sectional portion of the axial section the camera is oriented in the video frame; and
incrementing a counter for the cross-sectional portion to which the camera is oriented;

when the camera has moved to the end of the axial section:
comparing the counter for each cross-sectional portion to an optimal value; and
visualizing the variance of the counter from the optimal value for each cross-sectional portion.

15. The apparatus of claim 14 wherein the cross-sectional portion of the axial section to which the camera is oriented is determined by a shadow area in the extracted video frame, wherein the cross-sectional portion to which the camera is oriented is the cross-sectional portion opposite the cross-sectional portion containing the shadow area.

16. The apparatus of claim 14 , wherein software performs the further functions of:
determining that the end of an axial section has been reached by estimating the distance that the camera has moved through the axial section and comparing it with the given length of the axial section, wherein estimating the distance that the camera has moved through the axial section between frames comprises:
calculating an optical flow comprising a vector pattern caused by relative motion between the camera and an interior surface of the colon as determined from successive extracted video frames;
projecting the vector pattern to an epi-polar plane; and
calculating an axial vector based on the projected vector pattern with respect to a focus of expansion.

17. The apparatus of claim 16 wherein the axial travel direction of the camera is determined from the direction of the optical flow vectors with respect to the focus of expansion and wherein the speed of the camera as it moves through the axial section is given by the magnitude of the axial vector.

18. The apparatus of claim 14 wherein the visualization comprises, for each cross-sectional portion in each axial section, a cell comprising a heat map indicating the variance of the counter for each cross-sectional portion, wherein the cells are arranged in a matrix of color rectangles wherein the rows of the matrix indicate the cross-sectional portions and the columns of the matrix indicate the axial sections.

19. The apparatus of claim 18 wherein additional rows in the matrix include information regarding other measurements for each axial section, wherein the other measurements include a clarity score, a colonoscopy preparation score, and a colon distention score.

Assignees

• Univ Carnegie Mellon
• Allegheny Singer Res Institute

Inventors

• Cai Yang
• Thakkar Shyam

Classifications

• H04N23/555

  for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes · CPC title

• H04N7/185

  from a mobile camera, e.g. for remote control · CPC title

• G06T2207/30168

  Image quality inspection · CPC title

• G06T2207/10068

  Endoscopic image · CPC title

• G06T7/0014

  using an image reference approach · CPC title