Method and apparatus for estimating area or volume of object of interest from gastrointestinal images

Patent metadata

Field	Value
Publication number	US-10580157-B2
Application number	US-201916416266-A
Country	US
Kind code	B2
Filing date	May 20, 2019
Priority date	Aug 4, 2017
Publication date	Mar 3, 2020
Grant date	Mar 3, 2020

Abstract

A method and apparatus for estimating or measuring a physical area or physical volume of an object of interest in one or more images captured using an endoscope are disclosed. According to the present method, an object of interest in an image or images is determined. Also, the endoscope captures one or more structured-light images to derive distance information associated with the object of interest with respect to the camera of the endoscope. The physical area size or physical volume size of the object of interest is then determined based on the image or images, and the distance information.

First claim

The invention claimed is:

1. A method of estimating a physical length, physical area or physical volume of an object of interest in a regular image captured using an endoscope, the method comprising:
receiving the regular image captured using the endoscope, wherein the regular image is captured by projecting regular light from the endoscope to a scene corresponding to a field of view of the endoscope when the endoscope is inside a human GI (gastrointestinal) tract;
receiving one or more structured-light images captured using the endoscope when the endoscope is inside the human GI tract, wherein said one or more structured-light images are temporally close to the regular image and each of said one or more structured-light images is captured by projecting a plurality of light beams from the endoscope to the field of view of the endoscope;
determining the object of interest in the regular image;
determining correspondence between the plurality of light beams and dot locations of the plurality of light beams in said one or more structured-light images;
determining 3D (three-dimensional) locations of light beams-object intersections for the object of interest based on triangulation of the plurality of light beams and camera rays of dot locations of the plurality of light beams in said one or more structured-light images; and
determining the physical length, the physical area or the physical volume of the object of interest based on the 3D locations of light beams-object intersections on the object of interest.

2. The method of claim 1 , wherein a target dot location of a target light beam in a corrected structured-light image is associated with a target 3D location of light beams-object intersections on the object of interest and the target dot location of the target light beam falls on an epipolar line, and wherein two points of the epipolar line correspond to two test dot locations of the target light beam in two corrected test structured-light images at a near test distance and a far test distance.

3. The method of claim 2 , wherein the target 3D location of light beams-object intersections on the object of interest is derived according to an intersection of a projector ray and a camera ray, and wherein the projector ray is associated with the target light beam and the camera ray is associated with camera geometry passing through the target dot location of the target light beam.

4. The method of claim 3 , wherein the target 3D location of light beams-object intersections on the object of interest is determined using triangulation to minimize a sum of first distance between the target 3D location of light beams-object intersections on the object of interest and the projector ray and second distance between the target 3D location of light beams-object intersections on the object of interest and the camera ray.

5. The method of claim 2 , wherein corresponding dot locations associated with each of the plurality of light beams are pre-computed for multiple selected distances of each epipolar line and the corresponding dot locations for each of the plurality of light beams are stored in a table.

6. The method of claim 5 , wherein if the target dot location of the target light beam is not at any pre-computed dot locations of the target light beam, the target 3D location of light beams-object intersections on the object of interest associated with the target dot location of the target light beam is interpolated from selected distances of two corresponding dot locations of the target light beam closest to the target dot location of the target light beam.

7. The method of claim 6 , wherein said two corresponding dot locations, p 1 and p 2 of the target light beam correspond to object-camera distances, P 1 and P 2 and the object-camera distance P for the target dot location, p of the target light beam is derived according to P=P 1 +w*(P 2 −P 1 ), and wherein w=|p−p 1 |/|p 2 −p 1 |.

8. The method of claim 1 , wherein a triangle mesh is generated to cover a topographic surface of the object of interest, and wherein the triangle mesh consists of a set of triangles generated based on the dot locations of the plurality of light beams in said one or more structured-light images.

9. The method of claim 8 , wherein the triangle mesh is subdivided into a denser and smoother mesh.

10. The method of claim 8 , wherein an outline is used to identify the object of interest and an area of a target set of triangles within the outline is determined as the physical area of the object of interest.

11. The method of claim 10 , wherein a vector sum of target triangles of the target set of triangles is calculated, and wherein each vector area associated with each target triangle corresponds to a product of area and normal vector of each target triangle.

12. A system for estimating or measuring a physical length, physical area or physical volume of an object of interest in one or more images captured using an endoscope, the system comprising:
an interface configured to:
receive a regular image captured using the endoscope, wherein the regular image is captured by projecting regular light from the endoscope to a scene corresponding to a field of view of the endoscope when the endoscope is inside a human GI (gastrointestinal) tract; and
receive one or more structured-light images captured using the endoscope when the endoscope is inside the human GI tract, wherein said one or more structured-light images are temporally close to the regular image and each of said one or more structured-light images is captured by projecting a plurality of light beams from the endoscope to the field of view of the endoscope;

a processor configured to:
determine the object of interest in said one or more images;
determine correspondence between the plurality of light beams and dot locations of the plurality of light beams in said one or more structured-light images;

determine 3D (three-dimensional) locations of light beams-object intersections for the object of interest based on triangulation of the plurality of light beams and camera rays of dot locations of the plurality of light beams in said one or more structured-light images; and
determine the physical length, the physical area or the physical volume of the object of interest based on the 3D locations of light beams-object intersections on the object of interest; and

a display device configured to:
display the regular image, said one or more structured-light images, information related to the regular image or said one or more structured-light images, user data or a combination thereof.

13. The system of claim 12 , further comprising a computer interface device to allow a user to indicate an outline of the object of interest.

14. A method of calibrating an endoscope, the method comprising:
placing a target endoscope in a first controlled environment;
capturing a first test regular-image and a second test regular-image using the target endoscope by placing a test cylinder with known contents at a first distance and a second distance respectively;
deriving camera rays for image pixels based on the first test regular-image, the second test regular-image and corresponding 3D (three-dimensional) locations of the test cylinder at the first distance and the second distance;
placing the target endoscope in a second controlled environment;
capturing a first test SL (structured-light) image and a second test SL image using the target endoscope by projecting a plurality of light beams onto a uniform-background cylinder at the first distance and the second distance respectively;
deriving projector rays for the plurality of light b

Assignees

• Capsovision Inc

Inventors

• Wang Kang-Huai
• Hadley Mark
• Wu Chenyu

Classifications

• A61B1/2736

  Gastroscopes · CPC title

• A61B1/041

  Capsule endoscopes for imaging · CPC title

• A61B1/00057

  provided with means for testing or calibration · CPC title

• G06T2207/10068

  Endoscopic image · CPC title

• G01B11/2513

  with several lines being projected in more than one direction, e.g. grids, patterns · CPC title