Whole slide image registration and cross-image annotation devices, systems and methods

Patent metadata

Field	Value
Publication number	US-9818190-B2
Application number	US-201414775582-A
Country	US
Kind code	B2
Filing date	Mar 12, 2014
Priority date	Mar 14, 2013
Publication date	Nov 14, 2017
Grant date	Nov 14, 2017

Abstract

The disclosure relates to devices, systems and methods for image registration and annotation. The devices include computer software products for aligning whole slide digital images on a common grid and transferring annotations from one aligned image to another aligned image on the basis of matching tissue structure. The systems include computer-implemented systems such as work stations and networked computers for accomplishing the tissue-structure based image registration and cross-image annotation. The methods include processes for aligning digital images corresponding to adjacent tissue sections on a common grid based on tissue structure, and transferring annotations from one of the adjacent tissue images to another of the adjacent tissue images.

First claim

The invention claimed is:

1. A computerized image registration process, comprising:
a. selecting a first digital image of a first tissue section from a set of digital images of adjacent tissue sections;
b. selecting a second digital image of a second tissue section from the set;
c. matching tissue structure between the first digital image and the second digital image, said matching tissue structure comprising:
a coarse registration mode comprising:
generating a first gray-level tissue foreground image from the first digital image and generating a second gray-level tissue foreground image from the second digital image; computing a first tissue binary edge map from the first gray-level tissue foreground image and computing a second tissue binary edge map from the second gray-level tissue foreground image; and

d. automatically mapping an annotation drawn on the first digital image to the second digital image.

2. A computerized image registration process according to claim 1 , wherein the first digital image is derived from an image obtained using a stain and an imaging mode, and the second digital image is derived from an image obtained using at least one of a different stain, and a different imaging, as compared to the first digital image.

3. A computerized image registration process according to claim 2 , wherein the stain is chosen from a hematoxylin and eosin stain (‘HE’ stain), ImmunoHistoChemistry stain (“IHC” stain) or Fluorescent stain.

4. A computerized image registration process according to claim 2 , wherein the imaging mode is chosen from brightfield microscopy and fluorescent microscopy.

5. A computerized image registration process according to claim 1 , wherein said course registration mode further comprising: computing global transformation parameters to align the first binary edge map and the second binary edge map; and, mapping the first digital image and the second digital image to a common big grid encompassing both the first and second digital images based on the global transformation parameters.

6. A computerized image registration process according to claim 5 , wherein computing global transformation parameters comprises using a moments-based mapping method to generate an affine mapping between the first binary edge map and the second binary edge map.

7. A computerized image registration process according to claim 5 , further comprising a fine registration mode to refine alignment of the first digital image and the second digital image.

8. A computerized image registration process according to claim 7 , wherein the fine registration mode comprises: annotating the first digital image; mapping the annotation on the common big grid to a corresponding location in the second digital image; and, updating the location using Chamfer-distance matching based on the binary tissue edge maps.

9. A computerized image registration process according to claim 8 , wherein cropped versions of the tissue edge binary maps are used and the method further comprises selecting a minimum cost window which improves matching relative to coarse mode registration.

10. An image registration system, comprising:
a. a processor;
b. a memory containing instructions for execution by the processor, which if executed results in one or more of: aligning one or more images of adjacent tissue sections based on matching tissue structure to result in a set of aligned images, wherein each of the one or more images is prepared using a different stain, a different imaging mode, or both; and, replicating an annotation made by a user on one of the aligned images on at least another of the aligned images, wherein aligning the one or more images comprises: computing a soft weighted foreground image from a digital image, extracting a binary tissue edge-map from the foreground image, computing global transformation parameters based on the tissue edge-map, and mapping the tissue edge-map to a common grid on the global transformation parameters;
c. a client user interface for triggering the processor to execute the instructions; and,
d. a monitor which can display the client user interface, the first image and the second image, the results and combinations thereof.

11. The image registration system according to claim 10 , implemented on a workstation comprising at least one of a computer comprising the processor, the memory, the client user interface, and the monitor.

12. The image registration system according to claim 10 , implemented on a computer network.

13. The image registration system according to claim 12 , wherein the computer network comprises one or more client computers, a server, and a network-accessible database, all connected via a network, wherein the one or more client computers comprise the processor, the monitor, and the client user interface; the network-accessible database stores at least one set of images of adjacent tissue sections; and, the memory resides on the server or one or more client computers or both.

14. The image registration system according to claim 10 , wherein aligning the one or more images further comprise: identifying a first region surrounding an annotation on a first aligned image, identifying a second region in a second aligned image, wherein the second region is larger than the first region and is co-located on common grid with the first region, optimizing the location of the first region in the second region using an iterative process based on distance transformation and minimum cost function calculations.

15. A computer program product for aligning images and mapping an annotation from one aligned image to another aligned image, the computer program product comprising: a tangible non-transitory computer readable storage medium comprising a computer readable program code which, when executed by a processor of a computer, causes the computer to match tissue structure between a first digital image from a set of digital images of adjacent tissue sections and a second digital image from the set of digital images of adjacent tissue sections, wherein each image in the set is obtained using a different stain, a different imaging mode, or both; and transfer an annotation from the first digital image to the second digital image based on the matched tissue structure,
wherein matching tissue structure comprises computing a soft weighted foreground image for each of the first and second digital image, extracting a binary tissue edge-map for each of the first and second foreground digital images, computing global transformation parameters based on the first and second tissue edge-maps, and mapping the first image and the second image to a common grid based on the global transformation parameters, wherein the common grid has a center, and the first foreground image and the second foreground image each have a center, and the first and second foreground images are laid on the common grid such that the center of the common grid coincides with the center of the first foreground image and the center of the second foreground image.

16. A computer program product according to claim 15 , wherein transferring an annotation comprises mapping an annotation on the first digital image to a corresponding location on the second image as determined by the common grid.

17. A computer program product according to claim 16 , wherein the transferring the annotation further comprises: adjusting the location of the annotation on the second image based on a fine registration process comprising defining a first window around the annotation on the first image; defining a second window around the annotation o

Assignees

• Ventana Med Syst Inc

Inventors

• Chukka Srinivas
• Sarkar Anindya
• Sabata Bikash
• Yuan Quan

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T7/0014 Primary

  using an image reference approach · CPC title

• G06K9/6201

  Physics · mapped topic

• G06K9/00127

  Physics · mapped topic

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title