Learning based approach for aligning images acquired with different modalities

Patent metadata

Field	Value
Publication number	US-10733744-B2
Application number	US-201815927011-A
Country	US
Kind code	B2
Filing date	Mar 20, 2018
Priority date	May 11, 2017
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

Methods and systems for aligning images for a specimen acquired with different modalities are provided. One method includes acquiring information for a specimen that includes at least first and second images for the specimen. The first image is acquired with a first modality different than a second modality used to acquire the second image. The method also includes inputting the information into a learning based model. The learning based model is included in one or more components executed by one or more computer systems. The learning based model is configured for transforming one or more of the at least first and second images to thereby render the at least the first and second images into a common space. In addition, the method includes aligning the at least the first and second images using results of the transforming. The method may also include generating an alignment metric using a classifier.

First claim

What is claimed is:

1. A system configured to align images for a specimen acquired with different modalities, comprising:
one or more computer subsystems configured for acquiring information for a specimen, wherein the information comprises at least first and second images for the specimen, wherein the first image is acquired with a first modality different from a second modality used to acquire the second image, and wherein one of the first and second modalities is a modality of an electron beam imaging system configured for generating the first or second images for the specimen by directing electrons to the specimen and detecting electrons from the specimen; and
one or more components executed by the one or more computer subsystems, wherein the one or more components comprise a learning based model, wherein the one or more computer subsystems are configured to input the information for the specimen into the learning based model, wherein the learning based model is configured for transforming one or more of the at least first and second images to thereby render the at least first and second images into a common space, and wherein the one or more components are configured for aligning the at least the first and second images using results of said transforming.

2. The system of claim 1 , wherein the first and second modalities generate the first and second images with different pixel sizes.

3. The system of claim 1 , wherein the first and second modalities generate the first and second images with different frequency spreads.

4. The system of claim 1 , wherein the first and second modalities generate the first and second images with different distortions of patterned features formed on the specimen.

5. The system of claim 1 , wherein the first and second modalities are different modalities of the same imaging system.

6. The system of claim 1 , wherein the first and second modalities are different modalities of the same type of imaging system.

7. The system of claim 1 , wherein the first and second modalities are modalities of different types of imaging systems.

8. The system of claim 1 , wherein the first modality comprises scanning electron microscopy, and wherein the second modality comprises computer aided design.

9. The system of claim 1 , wherein the first modality comprises broadband optical imaging, and wherein the second modality comprises scanning electron microscopy.

10. The system of claim 1 , wherein the information further comprises at least a third image for the specimen, wherein the third image is acquired with a third modality different from the first and second modalities, wherein the one or more components further comprise an additional learning based model, wherein the additional learning based model is configured for transforming one or more of the at least first, second, and third images to thereby render two or more of the at least first, second, and third images into a common space, wherein the one or more components are further configured for aligning the two or more of the at least first, second, and third images using results of said transforming performed by the additional learning based model, wherein one of the first, second, and third modalities comprises broadband optical imaging, and wherein another of the first, second, and third modalities comprises computer aided design.

11. The system of claim 1 , wherein the information further comprises at least a third image for the specimen, wherein the third image is acquired with a third modality different from the first and second modalities, wherein the one or more components further comprise an additional learning based model, wherein the additional learning based model is configured for transforming one or more of the at least first, second, and third images to thereby render two or more of the at least first, second, and third images into a common space, wherein the one or more components are further configured for aligning the two or more of the at least first, second, and third images using results of said transforming performed by the additional learning based model, wherein one of the first, second, and third modalities comprises laser scanning optical imaging, and wherein another of the first, second, and, third modalities comprises broadband optical imaging.

12. The system of claim 1 , wherein the information further comprises at least a third image for the specimen, wherein the third image is acquired with a third modality different from the first and second modalities, wherein the one or more components further comprise an additional learning based model, wherein the additional learning based model is configured for transforming one or more of the at least first, second, and third images to thereby render two or more of the at least first, second, and third images into a common space, wherein the one or more components are further configured for aligning the two or more of the at least first, second, and third images using results of said transforming performed by the additional learning based model, wherein one of the first, second, and third modalities comprises laser scanning optical imaging, and wherein another of the first, second, and third modalities comprises computer aided design.

13. The system of claim 1 , Wherein the information further comprises at least a third image for the specimen, wherein the third image is acquired with a third modality different from the first and second modalities, wherein the one or more components further comprise an additional learning based model, wherein the additional learning based model is configured for transforming one or more of the at least first, second, and third images to thereby render two or more of the at least first, second, and third images into a common space, wherein the one or more components are further configured for aligning the two or more of the at least first, second, and third images using results of said transforming performed by the additional learning based model, wherein one of the first, second, and third modalities comprises low resolution optical imaging, and wherein another of the first, second, and third modalities comprises computer aided design.

14. The system of claim 1 , wherein the common space is an image space.

15. The system of claim 1 , wherein the common space is a feature space.

16. The system of claim 1 , wherein the one or more components are further configured for performing said aligning without using a learning based technique.

17. The system of claim 1 , wherein the learning based model comprises a regression model.

18. The system of claim 17 , wherein the regression model comprises an autoencoder variant, a conditional generative adversarial network, or a denoise convolutional autoencoder.

19. The system of claim 1 , wherein the learning based model included in the one or more components is further configured for performing said aligning.

20. The system of claim 19 , wherein the learning based model comprises a first encoder into which the first image is input to thereby generate deep learning based features of the first image and a second encoder into which the second image is input to thereby generate deep learning based features of the second image, wherein the first and second encoders are followed by a concatenation layer into which the deep learning based features of the first and second images are input, and wherein the concatenation layer is followed by one or more fully, connected layers configured for performing said aligning.

21

Assignees

• Kla Tencor Corp

Inventors

• Ha Thanh Huy
• Young Scott A
• Mahadevan Mohan

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06F18/2135

  based on approximation criteria, e.g. principal component analysis · CPC title

• G06F18/253

  of extracted features · CPC title

• G06F18/24

  Classification techniques · CPC title