Method and system for surgical tool localization during anatomical surgery

Patent metadata

Field	Value
Publication number	US-9905000-B2
Application number	US-201514860914-A
Country	US
Kind code	B2
Filing date	Sep 22, 2015
Priority date	Feb 19, 2015
Publication date	Feb 27, 2018
Grant date	Feb 27, 2018

Abstract

Various aspects of a method and system to localize surgical tools during anatomical surgery are disclosed herein. In accordance with an embodiment of the disclosure, the method is implementable in an image-processing engine, which is communicatively coupled to an image-capturing device that captures one or more video frames. The method includes determination of one or more physical characteristics of one or more surgical tools present in the one or more video frames, based on one or more color and geometric constraints. Thereafter, two-dimensional (2D) masks of the one or more surgical tools are detected, based on the one or more physical characteristics of the one or more surgical tools. Further, poses of the one or more surgical tools are estimated, when the 2D masks of the one or more surgical tools are occluded at tips and/or ends of the one or more surgical tools.

First claim

What is claimed is:

1. A system for surgical tool localization in an anatomical surgery, said system comprising:
one or more circuits in an image-processing engine communicatively coupled to an image-capturing device, wherein said image-capturing device is configured to capture at least one video frame,
wherein said one or more circuits are configured to:
detect two-dimensional (2D) masks of a plurality of surgical tools in said at least one video frame, based on color constraints and geometric constraints associated with said plurality of surgical tools; and
estimate poses of said plurality of surgical tools in said at least one video frame based on occlusion of said 2D masks of said plurality of surgical tools at one of tips or ends of said plurality of surgical tools.

2. The system of claim 1 , wherein said one or more circuits are further configured to remove, at least one smoke block from said at least one video frame to generate a smoke-free video frame, based on said occlusion of said at least one video frame is occluded with said at least one smoke block.

3. The system of claim 2 , wherein said one or more circuits are further configured to detect at least one smoke region in each of a set of video frames prior to said at least one video frame.

4. The system of claim 3 , wherein said at least one smoke block is removed from said at least one video frame based on an accumulated intensity of a set of pixels in said detected at least one smoke region in each of said set of video frames prior to said at least one video frame.

5. The system of claim 2 , wherein said one or more circuits are further configured to evolve a contour by contour evolution based on color characteristics of said plurality of surgical tools in said detected 2D masks of said plurality of surgical tools in said smoke-free video frame.

6. The system of claim 5 , wherein said contour evolution is based on a curvature and an intensity variance of regions one of inside or outside said contour of said detected 2D masks in said smoke-free video frame.

7. The system of claim 5 , wherein said one or more circuits are further configured to segment said smoke-free video frame to detect said plurality of surgical tools in said smoke-free video frame, based on said contour evolution.

8. The system of claim 1 , wherein said one or more circuits are further configured to filter color by an adaptive color-filtering process corresponding to said color constraints based on intensity of pixels in each region of said at least one video frame.

9. The system of claim 8 , wherein said adaptive color-filtering process is based on an opp2-color intensity-based filtration or a normalized opp2-color intensity-based filtration, on said each region of said at least one video frame.

10. The system of claim 8 , wherein said detection of said 2D masks of said plurality of surgical tools present in said at least one video frame is based on said adaptive color-filtering process.

11. The system of claim 1 , wherein said one or more circuits are further configured to estimate locations of said one of said tips or said ends of said plurality of surgical tools in said at least one video frame, based on an analysis of said detected 2D masks of said plurality of surgical tools along a first directional vector that extends along a longitudinal principal axis of said detected 2D masks of said plurality of surgical tools.

12. The system of claim 11 , wherein said one or more circuits are further configured to determine a centroid of said locations of said one of said tips or said ends of said plurality of surgical tools in said at least one video frame.

13. The system of claim 12 ,
wherein said one or more circuits are further configured to re-estimate said locations of said one of said tips or said ends of said plurality of surgical tools in said at least one video frame, based on said analysis of said detected 2D masks of said plurality of surgical tools,
wherein said re-estimation is along a second directional vector at a determined angle with respect to said first directional vector, and
wherein said locations of said one of said tips or ends are re-estimated based on said centroid that lies outside said detected 2D masks of said plurality of surgical tools, or said centroid that is occluded.

14. The system of claim 12 , wherein said estimation of said poses of said plurality of surgical tools is based on said estimated locations of said tips of said plurality of surgical tools based on said occlusion of one of said centroid or said ends of said plurality of surgical tools are occluded.

15. The system of claim 11 , wherein said estimation of said poses of said plurality of surgical tools is based on said estimated locations of said ends of said plurality of surgical tools based on said occlusion of said tips of said plurality of surgical tools.

16. The system of claim 1 , wherein a plurality of image-capture settings of said image-capturing device are adjusted based on said estimation of said poses of said plurality of surgical tools in said at least one video frame.

17. The system of claim 16 , wherein said plurality of image-capture settings comprise at least one of: an auto-exposure, an auto-focus, an auto-white-balance, or an auto-illumination.

18. The system of claim 1 , wherein said one or more circuits are further configured to display said at least one video frame via a user interface at a time of said anatomical surgery, wherein said plurality of surgical tools are one of masked or highlighted in said displayed at least one video frame.

19. The system of claim 1 , wherein said one or more circuits are further configured to generate a notification indicative of said occlusion of said plurality of surgical tools at said one of said tips or said ends of said plurality of surgical tools, wherein said notification corresponds to at least one of an audio alert, a textual alert, a visual alert, or a haptic alert.

20. A method for surgical tool localization during anatomical surgery, said method comprising:
in an image-processing engine communicatively coupled to an image-capturing device said image-capturing device configured to capture at least one video frame:
detecting two-dimensional (2D) masks of a plurality of surgical tools present in said at least one video frame, based on color constraints and geometric constraints associated with said plurality of surgical tools; and
estimating poses of said plurality of surgical tools in said at least one video frame based on occlusion of said 2D masks of said plurality of surgical tools at one of tips or ends of said plurality of surgical tools.

21. The method of claim 20 , further comprising estimating locations of said one of said tips or said ends of said plurality of surgical tools in said at least one video frame, based on an analysis of said detected 2D masks of said plurality of surgical tools along a first directional vector that extends along a longitudinal principal axis of said detected 2D masks of said plurality of surgical tools.

22. The method of claim 21 , further comprising determining a centroid of said locations of said one of said tips or said ends of said plurality of surgical tools in said at least one video frame.

23. The method of claim 22 , further comprising re-estimating said locations of said one of said tips or said ends of said plurality of surgical tools in said at least one video frame, based on said analysis of said detected 2D masks of said

Assignees

• Sony Corp

Inventors

• Chou Chen Rui
• Liu Ming Chang

Classifications

• G06T7/11

  Region-based segmentation · CPC title

• A61B34/20

  Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis · CPC title

• G06T5/20

  using local operators · CPC title

• G06T7/73

  using feature-based methods · CPC title

• A61B2090/3762

  using computed tomography systems [CT] · CPC title