Image rotation in an endoscopic hyperspectral, fluorescence, and laser mapping imaging system

Patent metadata

Field	Value
Publication number	US-12064088-B2
Application number	US-202117493708-A
Country	US
Kind code	B2
Filing date	Oct 4, 2021
Priority date	Jun 20, 2019
Publication date	Aug 20, 2024
Grant date	Aug 20, 2024

Abstract

Image rotation in an endoscopic hyperspectral, fluorescence, and/or laser mapping imaging system is described. A system includes an emitter for emitting pulses of electromagnetic radiation and an image sensor comprising a pixel array for sensing reflected electromagnetic radiation. The system includes a rotation sensor for detecting an angle of rotation of a lumen relative to a handpiece of an endoscope. The system is such that at least a portion of the pulses of electromagnetic radiation emitted by the emitter comprises one or more of a hyperspectral emission, a fluorescence emission, and/or a laser mapping pattern.

First claim

What is claimed is:

1. A system comprising:
an endoscope;
a handpiece unit;
an emitter for emitting pulses of electromagnetic radiation, comprising a plurality of electromagnetic sources, the sources comprising:
a visible source for pulsing a visible wavelength of electromagnetic radiation, and
a specialty source comprising a mapping source and one or more of:
a fluorescence source for pulsing a fluorescence excitation wavelength of electromagnetic radiation, or
a spectral source for pulsing electromagnetic radiation within a waveband comprising one or more wavelengths selected to elicit a spectral response from a tissue;

one or more image sensors each comprising a pixel array for sensing electromagnetic radiation; and
a controller in electronic communication with the emitter and the one or more image sensors;
a rotation sensor for sensing rotation of the handpiece unit and/or the endoscope;
wherein the one or more image sensors sense reflected electromagnetic radiation in response to pulses of reflected electromagnetic radiation from the emitter to generate a plurality of exposure frames, and wherein the plurality of exposure frames comprises a mapping exposure frame and one or more of a spectral exposure frame or a fluorescence exposure frame.

2. The system of claim 1 , wherein the mapping exposure frame comprises data for calculating one or more of a topography of a scene, a dimension of one or more objects within the scene, a location of one or more tools within the scene, or a distance between the one or more objects and the endoscope.

3. The system of claim 1 , wherein the spectral exposure frame is sensed in response to a multispectral emission pulsed by the spectral source, and wherein the multispectral emission comprises electromagnetic radiation comprising a wavelength within a multispectral range, wherein the multispectral range comprises one or more of:
wavelengths from about 510 nm to about 590 nm; or
wavelengths from about 900 nm to about 1000 nm.

4. The system of claim 1 , wherein the fluorescence exposure frame is sensed in response to a fluorescence excitation emission pulsed by the fluorescence source, and wherein the fluorescence excitation emission causes a reagent to fluoresce.

5. The system of claim 4 , wherein the fluorescence excitation emission comprises electromagnetic radiation comprising a wavelength within a fluorescence range, wherein the fluorescence range comprises one or more of:
wavelengths from about 770 nm to about 795 nm; or
wavelengths from about 790 nm to about 815 nm.

6. The system of claim 1 , wherein:
the controller synchronizes timing of the emitter and the one or more image sensors such that the one or more image sensors sense the plurality of frames in response to the emitter pulsing the plurality of emissions of electromagnetic radiation; and
wherein the controller is configured to execute instructions, wherein the instructions comprise providing the mapping exposure frame to a corresponding laser mapping system configured to calculate one or more of a topography of a scene, a dimension of one or more objects within the scene, a location of one or more tools within the scene, or a distance between the one or more objects and the endoscope based on the mapping exposure frame.

7. The system of claim 6 , wherein the instructions executed by the controller further comprise one or more of:
providing the spectral exposure frame to a corresponding multispectral system configured to identify one or more tissue structures within the scene based on data from the multispectral frame; or
providing the fluorescence exposure frame to a corresponding fluorescence system configured to identify a location of a reagent within the scene based on data from the fluorescence frame.

8. The system of claim 7 , wherein the plurality of frames sensed by the one or more image sensors further comprises a color image frame sensed in response to an emission of visible electromagnetic radiation, and wherein the instructions executed by the controller further comprise:
generating an overlay frame comprising the color image frame, data extracted from the laser mapping frame, and one or more of:
an indication of the one or more tissue structures within the scene as determined based on the multispectral frame; or
an indication of the location of the reagent within the scene as determined based on the fluorescence frame; and

providing the overlay frame to a display for real-time visualization of the scene.

9. The system of claim 1 , wherein the rotation sensor comprises one or more of a rotation-detecting Hall-effect sensor, a diametrically-polarized magnetic annulus, a gyroscope, or a potentiometer.

10. The system of claim 1 , further comprising one or more processors for executing an image signal processing pipeline comprising instructions for generating output image frames, wherein the instructions comprise:
receiving an angle of rotation for rotating at least one frame of the plurality of frames to maintain a constant image horizon, wherein the angle of rotation is determined based on output from the rotation sensor;
identifying integer coordinates for pixel data in the at least one frame of the plurality of frames;
applying a rotation kernel to the integer coordinates to transform the integer coordinates to real-number pixel coordinates.

11. The system of claim 10 , wherein the instructions for the image signal processing pipeline further comprise:
identifying one or more pixels within the at least one frame of the plurality of frames that is void after applying the rotation kernel; and
filling the void one or more pixels by applying nearest neighbor substitution, bilinear interpolation, or bicubic interpolation.

12. The system of claim 1 , further comprising one or more processors for executing an image signal processing pipeline comprising instructions for generating output image frames, wherein the instructions comprise:
receiving an angle of rotation from the rotation sensor, wherein the rotation sensor senses the angle of rotation of the handpiece unit relative to the endoscope;
determining whether to apply image transformation to maintain a constant image horizon based on whether the same angle of rotation is sensed by the rotation sensor across a threshold number of frames; and
in response to the same angle of rotation being sensed across the threshold number of frames, applying the image transformation to at least one frame of the plurality of frames to maintain the constant image horizon.

13. The system of claim 1 , wherein the rotation sensor senses an angle of rotation of the handpiece unit relative to the endoscope, and wherein the one or more image sensors are disposed at a distal end of the endoscope.

14. The system of claim 1 ,
wherein the visible source comprises one or more of a white source for emitting white electromagnetic radiation, a red source for emitting red electromagnetic radiation, a blue source for emitting blue electromagnetic radiation, or a green source for emitting green electromagnetic radiation; and
wherein the mapping source emits one or more of vertical hashing, horizontal hashing, a raster grid of discrete points, an occupancy grid map, or a dot array.

15. The system of claim 14 , wherein:
the spectral source comprises one or more multispectral sources for emitting electromagnetic radiation comprising a wavelength within a multispectral range, wherein the multispectral range comprises one or more of:
wavelengths from about 510 nm to about 590 nm; or
wavelengths from about 900 nm to about 1000 nm; and

the fluo

Assignees

• Cilag Gmbh Int
• Cllag Gmbh Int

Inventors

• Talbert Joshua D
• Wichern Donald M

Classifications

• H04N23/555

  for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes · CPC title

• H04N23/84

  for processing colour signals · CPC title

• H04N23/72

  Combination of two or more compensation controls · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/56

  provided with illuminating means · CPC title