Method and apparatus for registration of different mammography image views

Patent metadata

Field	Value
Publication number	US-11657497-B2
Application number	US-202016829556-A
Country	US
Kind code	B2
Filing date	Mar 25, 2020
Priority date	Mar 26, 2019
Publication date	May 23, 2023
Grant date	May 23, 2023

Abstract

A method of identifying potential lesions in mammographic images may include operations executed by an image processing device including receiving first image data of a first type, receiving second image data of a second type, registering the first image data and the second image data by employing a CNN using pixel level registration or object level registration, determining whether a candidate detection of a lesion exists in both the first image data and the second image data based on the registering of the first image data and the second image data, and generating display output identifying the lesion.

First claim

That which is claimed:

1. A method of identifying potential lesions in mammographic images, the method comprising:
receiving, by an image processing device, first image data;
receiving, by the image processing device, second image data, one of the first image data or the second image data being two-dimensional Craniocaudal (CC) mammographic image data or two-dimensional Mediolateral Oblique (MLO) mammographic image data;
registering, by the image processing device, the first image data and the second image data by employing an image registration convolutional neural network (CNN) using pixel level registration; wherein registering the first image data with the second image data comprises:
inputting the first image data and the second image data into the image registration CNN;
generating, via convolutions performed by the image registration CNN on the first image data and the second image data, a deformation field of deformation vectors that map pixels of the first image data to pixels of the second image data; the deformation field comprising, to define the deformation vectors,
a vertical deformation data array that defines row-wise relationships between the pixels of the first image data and the pixels of the second image data and
a horizontal deformation data array that defines column-wise relationships between the pixels of the first image data and the pixels of the second image data;

determining, by the image processing device, whether a candidate detection of a lesion exists in both the first image data and the second image data based on the first image data and the second image data and a mapping of the first image data to the second image data provided by deformation field output from the image registration CNN; and
generating, by the image processing device, display output identifying the lesion.

2. The method of claim 1 , wherein the first image data is two-dimensional Craniocaudal (CC) mammographic image data and the second image data is two-dimensional Mediolateral Oblique (MLO) mammographic image data.

3. The method of claim 2 , wherein generating the deformation field comprises:
learning a mapping from a first image of the first image data to a second image of the second image data via the CNN; and
generating a warped image output based on the mapping.

4. The method of claim 3 , wherein determining whether the candidate detection exists in both the first image data and the second image data comprises analyzing the warped image output for correlated features associated with the candidate detection, and
wherein generating the display output comprises generating an object identification on the warped image.

5. The method of claim 3 , wherein the CNN is a fully convolutional network,
wherein the CNN does not include any fully connected layer, and
wherein the first image data and the second image data are input to the CNN in separate channels such that an output of the CNN comprises two channels defining the vertical deformation data array and the horizontal deformation data array.

6. The method of claim 3 , wherein the CNN comprises a skip architecture including one or more skip paths, and
wherein each of the one or more skip paths is implemented to generate the deformation field.

7. The method of claim 2 , further comprising performing object level registration, wherein performing object level registration comprises:
employing a first stage CNN architecture configured to independently analyze the first image data and the second image data to identify candidate regions;
conducting pairwise evaluation of the candidate regions to determine whether the candidate detection exists; and
employing a second stage CNN architecture configured to determine candidate matches and generate object identification based on the pairwise evaluation.

8. The method of claim 7 , wherein employing the first stage CNN architecture comprises:
providing a plurality of first images associated with the first image data to a first region-based CNN (R-CNN) trained to identify first candidate regions; and
providing a plurality of second images associated with the second image data to a second R-CNN trained to identify second candidate regions,
wherein the first and second candidate regions comprise the candidate regions on which the pairwise evaluation is conducted.

9. The method of claim 8 , wherein employing the second stage CNN architecture comprises providing the second stage CNN with data associated with the first candidate regions and the second candidate regions, and distance from nipple information for each instance of the candidate detection in the data associated with the first candidate regions and the second candidate regions.

10. The method of claim 9 , wherein generating the display output comprises generating an object identification on both the first images and the second images.

11. The method of claim 2 , further comprising utilizing both the pixel level registration to generate a first object identification and an object level registration to generate a second object identification, and comparing the first and second object identifications.

12. A method of identifying potential lesions in mammographic images, the method comprising:
receiving, by an image processing device, first image data of a first type;
receiving, by the image processing device, second image data of a second type;
learning, by the image processing device, a mapping from a first image of the first image data to a second image of the second image data by employing an image registration convolutional neural network (CNN) to perform pixel level registration between the first image data and the second image data, wherein registering the first image data with the second image data comprises:
inputting the first image data and the second image data into the image registration CNN;
generating, via convolutions performed by the image registration CNN on the first image data and the second image data, a deformation field of deformation vectors that map pixels of the first image data to pixels of the second image data; the deformation field comprising, to define the deformation vectors,
a vertical deformation data array that defines row-wise relationships between the pixels of the first image data and the pixels of the second image data and
a horizontal deformation data array that defines column-wise relationships between the pixels of the first image data and the pixels of the second image data;

generating, by the image processing device as an output of the image registration CNN based on the deformation field, a warped image output of the first image data mapped to the second image data using the deformation field;
determining, by the image processing device, whether a candidate detection of a lesion exists in both the first image data and the second image data based on the warped image; and
generating, by the image processing device, display output identifying the lesion.

13. The method of claim 12 , wherein the first image data is two-dimensional Craniocaudal (CC) mammographic image data and the second image data is two-dimensional Mediolateral Oblique (MLO) mammographic image data.

14. The method of claim 13 , wherein determining whether the candidate detection exists in both the first image data and the second image data comprises analyzing the warped image output for correlated features associated with the candidate detection, and
wherein generating the display output comprises generating an object identification on the warped image.

15. The method of claim 13 , wherein the CNN is a fully convolutional networ

Assignees

• Univ Johns Hopkins

Inventors

• Walton William C
• Kim Seung-Jun

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/30

  Determination of transform parameters for the alignment of images, i.e. image registration · CPC title

• A61B6/502

  for diagnosis of breast, i.e. mammography · CPC title

• G06T2207/30096

  Tumor; Lesion · CPC title

• G06T7/0012 Primary

  Biomedical image inspection · CPC title