Image processing device, endoscopic surgery system, and image processing method

Patent metadata

Field	Value
Publication number	US-11266294-B2
Application number	US-201716080120-A
Country	US
Kind code	B2
Filing date	Feb 21, 2017
Priority date	Mar 9, 2016
Publication date	Mar 8, 2022
Grant date	Mar 8, 2022

Abstract

Provided is an image processing device including: an image processing unit that generates image data for output from a captured image of an inside of a body cavity of a patient photographed by an endoscope. The image processing unit generates the image data in a manner that, when a lens barrel of the endoscope is moved in an optical axis direction of an objective lens during photographing, a display range in the captured image that is a range expressed in a display image displayed on a display device does not change, during the movement and after the movement, from a display range before the movement.

First claim

The invention claimed is:

1. An image processing device, comprising:
circuitry configured to:
generate first image data from a first image captured by an endoscope, wherein
the endoscope is at a first position, and
the first image is an image of an inside of a body of a patient;

control display of the generated first image data on a display device;
crop an area from a second image captured by the endoscope, wherein
the second image is captured after movement of the endoscope from the first position to a second position,
the cropped area corresponds to a display range of the displayed first image data,
the second image has a photographic range greater than the first image, and
a center of the cropped area is same as a center of the second image;

enlarge the cropped area;
generate second image data based on the enlarged cropped area; and
control display of the generated second image data on the display device, wherein the display range of the displayed first image data is substantially same as a display range of the displayed second image data.

2. The image processing device according to claim 1 , wherein the first image is a video image.

3. The image processing device according to claim 1 , wherein the circuitry is further configured to control movement of the endoscope in an optical axis direction of an objective lens of the endoscope.

4. An endoscopic surgery system, comprising:
an endoscope configured to:
capture a first image at a first position, wherein the first image is an image of an inside of a body of patient; and
capture a second image at a second position based on movement of the endoscope from the first position to the second position, wherein
the second image has a photographic range greater than the first image, and
the endoscope includes a lens barrel;

an image processing device including circuitry configured to:
generate first image data from the first image;
control display of the generated first image data on a display device;
crop an area from the second image, wherein
the cropped area corresponds to a display range of the displayed first image data, and
a center of the cropped area is same as a center of the second image;

enlarge the cropped area;
generate second image data based on the enlarged cropped area; and
control display of the generated second image data on the display device, wherein the display range of the displayed first image data is substantially same as a display range of the displayed second image data; and

an arm control device configured to control movement of an arm unit and the lens barrel, wherein
the circuitry is further configured to:
issue, based on a retraction condition, a first instruction to the arm control device to execute a retraction operation to move the lens barrel of the endoscope in a retraction direction from the first position to the second position, wherein
the retraction direction is an optical axis direction of an objective lens of the lens barrel, and
the second position is away from an observation target site than the first position; and

issue, based a return condition, a second instruction to the arm control device to execute a return operation to move the lens barrel in a return direction from the second position to the first position, wherein
the return direction is the optical axis direction of the objective lens, and
the lens barrel is movable from the second position to the first position near the observation target site.

5. The endoscopic surgery system according to claim 4 , wherein
the retraction condition is detection of mist from the first image, and
the return condition is a detection of at least a partial disappearance of the mist from the second image.

6. The endoscopic surgery system according to claim 5 , wherein, when generation of the mist is detected, a leading end of an energy device is detected as being present in the captured image, and the generation of the mist is detected in a predetermined range about the detected leading end of the energy device.

7. The endoscopic surgery system according to claim 4 , wherein the retraction condition is that mist presence areas occupy a first predetermined ratio or more of the captured image, and the return condition is that the mist presence areas occupy a second predetermined ratio or less of the captured image.

8. The endoscopic surgery system according to claim 4 , wherein the retraction condition is that a distance between a treatment instrument and a treatment site is a first predetermined threshold value or shorter, and the return condition is that the distance between the treatment instrument and the treatment site is a second predetermined threshold value or longer.

9. The endoscopic surgery system according to claim 4 , wherein the retraction condition is that an energy device is energized, and the return condition is that output of the energy device has stopped and at least partial disappearance of mist is detected based on the captured image.

10. The endoscopic surgery system according to claim 4 , wherein the retraction condition is that manipulation in which an operator performs treatment with an energy device is made, and the return condition is that manipulation in which the operator ends the treatment with the energy device is made.

11. The endoscopic surgery system according to claim 4 , wherein the retraction condition is that a grip mechanism at a leading end of a treatment instrument is detected to have gripped a part of a biological tissue based on the captured image, and the return condition is that the grip mechanism at the leading end of the treatment instrument is detected to have released the part of the biological tissue based on the captured image and disappearance of mist is detected based on the captured image.

12. The endoscopic surgery system according to claim 4 , wherein the retraction condition and the return condition include detection of a user-initiated instruction.

13. The endoscopic surgery system according to claim 4 , wherein the retraction condition is that a distance between a leading end of the endoscope and a treatment instrument is a first predetermined threshold value or shorter, and the return condition is that the distance between the leading end of the endoscope and the treatment instrument is a second predetermined threshold value or longer.

14. The endoscopic surgery system according to claim 4 , wherein a movement amount of the lens barrel at a time of the return operation is a same value as a movement amount of the lens barrel at a time of the retraction operation.

15. The endoscopic surgery system according to claim 4 , wherein a movement amount of the lens barrel at a time of the return operation is a value obtained by subtracting a distance between a leading end of the lens barrel and an observation target site immediately before generation of mist is detected, from a distance between the leading end of the lens barrel and the observation target site at which the mist does not obscure the captured image.

16. The endoscopic surgery system according to claim 4 , wherein the arm control device is further configured to:
control rotation of a plurality of joints of the arm unit in a coordinated manner; and
execute, based on the rotation of the plurality of joints, the retraction operation and the return operation.

17. The endoscopic surgery system according to claim 4 , further comprising:
a cylindrical holding unit, wherein
the lens barrel is inserted into the cylindrical holding unit,
the cylindrical holding unit is at a leading end

Assignees

• Sony Corp

Inventors

• Hirayama Tomoyuki
• Kikuchi Daisuke
• Sugie Yuki
• Matsuda Yasuhiro
• Nagao Daisuke
• Kasai Takara

Classifications

• A61B1/00009

  of image signals during a use of endoscope · CPC title

• A61B1/127

  with means for preventing fogging (A61B1/253 takes precedence) · CPC title

• G03B13/36

  Autofocus systems · CPC title

• A61B18/1442

  Probes having pivoting end effectors, e.g. forceps · CPC title

• A61B90/361

  Image-producing devices, e.g. surgical cameras · CPC title