Image processing device, method, and medium for calculating propagation speed and direction of object pulsations

Patent metadata

Field	Value
Publication number	US-11087472-B2
Application number	US-201815909456-A
Country	US
Kind code	B2
Filing date	Mar 1, 2018
Priority date	Jul 19, 2013
Publication date	Aug 10, 2021
Grant date	Aug 10, 2021

Abstract

An image processing device includes a processor to detect motion of objects in an observation area on the basis of a captured image. The processor also calculates amounts of each detected motion, and calculates a propagation speed, a propagation direction, or both, of the motion for pulsations of the objects.

First claim

The invention claimed is:

1. An image processing system, comprising:
a non-transitory computer-readable medium including computer-readable instructions that, when executed by processor circuitry, cause the processor circuitry to perform a method comprising:
detecting motion of objects in an observation area based on a captured image;
calculating amounts of each detected motion;
calculating a propagation speed and a propagation direction of the motion for each pulsation of a plurality of pulsations of the objects based on a variation over time of each calculated amount of motion;
calculating a stability of the plurality of pulsations based on a sum of (i) an average of the propagation directions for the plurality of pulsations multiplied by a first predetermined weight coefficient, and (ii) an average of the propagation speeds for the plurality of pulsations multiplied by a second predetermined weight coefficient different from the first weighted coefficient; and
controlling a display to display a waveform of the plurality of pulsations, a plurality of indications and an evaluation value corresponding to the stability, wherein

each indication of the plurality of indications is displayed alongside a corresponding pulsation of the plurality of pulsations, and
each indication is oriented in the propagation direction and has a length corresponding to the propagation speed of the pulsation.

2. The image processing system according to claim 1 , wherein the non-transitory computer-readable medium further includes computer-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to control the display to display both the propagation direction for each pulsation and the waveform representing the variation over time of each calculated amount of motion.

3. The image processing system according to claim 1 , wherein each indication is an arrow representing the propagation direction and propagation speed of the corresponding pulsation.

4. The image processing system according to claim 1 , wherein the non-transitory computer-readable medium further includes computer-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to detect the motion of the objects for each of a plurality of partial areas in the observation area on the captured image.

5. The image processing system according to claim 1 , wherein the object is selected from the group consisting of cultured myocardial cells, non-cultured myocardial cells, or a heart.

6. The image processing system according to claim 1 , wherein the non-transitory computer-readable medium further includes computer-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to
detect the motion for each of a plurality of partial areas in each frame of a moving image in the observation area, and
generate waveform information representing pulsations of the objects in the respective partial areas on the basis of the amounts of motion for each partial area in a plurality of frames of the moving image.

7. The image processing system according to claim 6 , wherein the non-transitory computer-readable medium further includes computer-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to calculate at least one of a propagation speed and the propagation direction for each pulsation by comparing a time corresponding to a feature point set in advance in a waveform of the waveform information in a predetermined partial area with a time corresponding to a feature point set in advance in a waveform of the waveform information in another partial area.

8. The image processing system according to claim 1 , wherein the non-transitory computer-readable medium further includes computer-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to calculate the stability of the plurality of pulsations in an entire observation area.

9. The image processing system according to claim 8 , wherein at least one of the following is also used in a weighted sum to determine the stability of pulsations:
a smallness of a difference between a maximum value and a minimum value of the propagation direction, and
a propagation speed of the pulsation.

10. The image processing system according to claim 8 , wherein the non-transitory computer-readable medium further includes computer-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to cause a transition over time of the evaluation value evaluating the stability of the pulsation to be displayed.

11. The image processing system according to claim 1 , wherein a pulsation of the object includes a contraction and an expansion of said object.

12. The image processing system according to claim 1 , wherein the non-transitory computer-readable medium further includes computer-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to control the display to display an indication of a peak of each pulsation on the waveform.

13. The image processing system according to claim 1 , wherein the non-transitory computer-readable medium further includes computer-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to:
detect motion of the objects, calculate the amounts of the motion, and calculate the propagation direction of the motion for a selected block of interest of the observation area of the captured image, and
control the display to display both the propagation direction for each pulsation and a waveform of the pulsations for the selected block of interest.

14. The image processing system according to claim 1 , wherein the non-transitory computer-readable medium further includes computer-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to:
divide the observation area of the captured image into a plurality of blocks, and
control the display to display an isochrone graph in which the plurality of blocks is displayed with colors based on a rise time of the waveform.

15. The image processing system according to claim 14 , wherein blocks of the plurality of blocks that have a same rise time are displayed with a same color.

16. The image processing system according to claim 1 , wherein the non-transitory computer-readable medium further includes computer-readable instructions that, when executed by the processor circuitry, cause the processor circuitry to control the display to display a speed of propagation for each pulsation.

17. The image processing system according to claim 1 , wherein the propagation direction for each pulsation is overlaid on the waveform of the pulsations.

18. The image processing system according to claim 1 , further comprising the processor circuitry.

19. An imaging system, comprising:
a microscope configured to capture an image of an observation area; and
processor circuitry configured to:
detect motion of objects in the observation area based on the image;
calculate amounts of each detected motion;
calculate a propagation speed and a propagation direction of the motion for each pulsation of a plurality of pulsations of the objects based on a variation over time of each calculated amount of motion;
calculate a stability of the plurality of pulsations based on a sum of (i) an average of the propagation directions for the plurality of pulsations multiplied by a first predete

Assignees

• Sony Corp

Inventors

• Kunihiro Takeshi
• Hayakawa Tomohiro
• Matsui Eriko
• Kobayashi Seiji

Classifications

• G06V20/69

  Microscopic objects, e.g. biological cells or cellular parts · CPC title

• G06T2207/10056

  Microscopic image · CPC title

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T2207/30024

  Cell structures in vitro; Tissue sections in vitro · CPC title