Fiber pattern removal and image reconstruction endoscopic devices and related methods

Patent metadata

Field	Value
Publication number	US-10891730-B1
Application number	US-201816206958-A
Country	US
Kind code	B1
Filing date	Nov 30, 2018
Priority date	Nov 30, 2017
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

Implementations of the present disclosure include methods, systems, and computer-readable storage mediums for image reconstruction. Actions include receiving an image acquired by an endoscopic system including an optical fiber bundle with multiple optical fibers, each optical fiber being surrounded by cladding, determining in the image fiber core locations corresponding to the optical fibers, reconstructing missing information from the image using interpolation performed in accordance with the fiber core locations, the missing information corresponding to artifacts in the acquired image that result from the cladding, and providing a fiber-pattern removed image, in which the artifacts in the acquired image are removed using the missing information.

First claim

What is claimed is:

1. A computer-implemented method for image reconstruction, the method being executed using one or more processors and comprising:
receiving, by the one or more processors, an image acquired by an endoscopic system comprising an optical fiber bundle comprising a plurality of optical fibers, wherein each of the plurality of optical fibers is surrounded by cladding;
determining, by the one or more processors, in the image a plurality of fiber core locations corresponding to the plurality of optical fibers in the optical fiber bundle;
reconstructing, by the one or more processors, missing information from the image using interpolation performed in accordance with the plurality of fiber core locations, wherein the missing information corresponds to artifacts in the acquired image that result from the cladding;
providing, by the one or more processors, a fiber-pattern removed image in which the artifacts in the acquired image are removed using the missing information; and
performing, by the one or more processors, spectral correction for the fiber-pattern removed image by correcting spectral distortions associated with the endoscopic system.

2. The method of claim 1 , wherein reconstructing the missing information comprises:
performing, by the one or more processors, Delaunay triangulation on the plurality of fiber core locations to generate triangulation data; and
performing, by the one or more processors, Barycentric interpolation using the triangulation data to produce the missing information.

3. The method of claim 2 , wherein determining the plurality of fiber core locations in the image comprises localizing fiber core pixels using a reference image and a two dimensional Gaussian function to model light coming out from each of the plurality of optical fibers.

4. The method of claim 2 , wherein determining the plurality of fiber core locations in the image comprises:
performing, by the one or more processors, a first round of fiber core pixel candidate screening using individual threshold values set in accordance with illumination settings of the endoscopic system; and
performing, by the one or more processors, a second round of fiber core pixel candidate screening to remove incorrect core pixels.

5. The method of claim 4 , wherein performing the second round of fiber core pixel candidate screening comprises using a mask-based approach to remove incorrect core pixels.

6. The method of claim 1 , wherein the image comprises a hyperspectral image.

7. The method of claim 6 , wherein performing the spectral correction comprises:
recovering same-bandwidth spectra.

8. The method of claim 6 , wherein performing the spectral correction comprises using a reshaped reflectance matrix.

9. The method of claim 1 , wherein the receiving, the determining, the reconstructing, and the providing are performed in semi-real time.

10. A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations for image reconstruction, the operations comprising:
receiving an image acquired by an endoscopic system comprising an optical fiber bundle comprising a plurality of optical fibers, wherein each of the plurality of optical fibers is surrounded by cladding;
determining in the image a plurality of fiber core locations corresponding to the plurality of optical fibers in the optical fiber bundle;
reconstructing missing information from the image using interpolation performed in accordance with the plurality of fiber core locations, wherein the missing information corresponds to artifacts in the acquired image that result from the cladding;
providing a fiber-pattern removed image in which the artifacts in the acquired image are removed using the missing information; and
performing spectral correction for the fiber-pattern removed image by correcting spectral distortions associated with the endoscopic system.

11. The computer-readable storage medium of claim 10 , wherein reconstructing the missing information comprises:
performing Delaunay triangulation on the plurality of fiber core locations to generate triangulation data; and
performing Barycentric interpolation using the triangulation data to produce the missing information.

12. The computer-readable storage medium of claim 11 , wherein determining the plurality of fiber core locations in the image comprises localizing fiber core pixels using a reference image and a two dimensional Gaussian function to model light coming out from each of the plurality of optical fibers.

13. The computer-readable storage medium of claim 11 , wherein determining the plurality of fiber core locations in the image comprises:
performing a first round of fiber core pixel candidate screening using individual threshold values set in accordance with illumination settings of the endoscopic system; and
performing a second round of fiber core pixel candidate screening to remove incorrect core pixels.

14. The computer-readable storage medium of claim 13 , wherein performing the second round of fiber core pixel candidate screening comprises using a mask-based approach to remove incorrect core pixels.

15. The computer-readable storage medium of claim 10 , wherein the image comprises a hyperspectral image.

16. A system, comprising:
an endoscopic system;
a client-side computing device; and
a computer-readable storage device coupled to the client-side computing device and having instructions stored thereon which are configured to cause the client-side computing device to perform operations for image reconstruction, the operations comprising:
receiving an image acquired by an endoscopic system comprising an optical fiber bundle comprising a plurality of optical fibers, wherein each of the plurality of optical fibers is surrounded by cladding,
determining in the image a plurality of fiber core locations corresponding to the plurality of optical fibers in the optical fiber bundle,
reconstructing missing information from the image using interpolation performed in accordance with the plurality of fiber core locations, wherein the missing information corresponds to artifacts in the acquired image that result from the cladding,
providing a fiber-pattern removed image in which the artifacts in the acquired image are removed using the missing information, and
performing spectral correction for the fiber-pattern removed image by correcting spectral distortions associated with the endoscopic system.

17. The system of claim 16 , wherein reconstructing the missing information comprises:
performing Delaunay triangulation on the plurality of fiber core locations to generate triangulation data; and
performing Barycentric interpolation using the triangulation data to produce the missing information.

18. The system of claim 17 , wherein determining the plurality of fiber core locations in the image comprises localizing fiber core pixels using a reference image and a two dimensional Gaussian function to model light coming out from each of the plurality of optical fibers.

19. The system of claim 17 , wherein determining the plurality of fiber core locations in the image comprises:
performing a first round of fiber core pixel candidate screening using individual threshold values set in accordance with illumination settings of the endoscopic system; and
performing a second round of fiber core pixel candidate screening to remove incorrect core pixels.

Assignees

• Univ Southern California

Inventors

• Wang Pu
• Cutrale Francesco
• Fraser Scott E

Classifications

• A61B1/000095 Primary

  for image enhancement · CPC title

• G06T7/0012 Primary

  Biomedical image inspection · CPC title

• G02B6/065

  with dynamic image improvement · CPC title

• A61B1/0017

  Details of single optical fibres, e.g. material or cladding · CPC title

• A61B1/00167

  Details of optical fibre bundles, e.g. shape or fibre distribution · CPC title