Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging

Patent metadata

Field	Value
Publication number	US-10169852-B1
Application number	US-201816027056-A
Country	US
Kind code	B1
Filing date	Jul 3, 2018
Priority date	Jul 3, 2018
Publication date	Jan 1, 2019
Grant date	Jan 1, 2019

Abstract

Systems, methods, and computer-readable media for feedback on and improving the accuracy of super-resolution imaging. In some embodiments, a low resolution image of a specimen can be obtained using a low resolution objective of a microscopy inspection system. A super-resolution image of at least a portion of the specimen can be generated from the low resolution image of the specimen using a super-resolution image simulation. Subsequently, an accuracy assessment of the super-resolution image can be identified based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens identified using a simulated image classifier. Based on the accuracy assessment of the super-resolution image, it can be determined whether to further process the super-resolution image. The super-resolution image can be further processed if it is determined to further process the super-resolution image.

First claim

What is claimed is:

1. A method for generating a super-resolution image for a specimen through a super-resolution system based on a low resolution image of the specimen comprising:
obtaining the low resolution image of the specimen using a low resolution objective of a microscopy inspection system;
generating the super-resolution image of at least a portion of the specimen from the low resolution image of the specimen using a super-resolution image simulation;
identifying an accuracy assessment of the super-resolution image based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens identified using a simulated image classifier;
determining whether to further process the super-resolution image based on the accuracy assessment of the super-resolution image;
obtaining one or more high resolution images of the at least the portion of the specimen using a high resolution objective of the microscopy inspection system, if it is determined to further process the super-resolution image based on the accuracy assessment of the super-resolution image;
assembling the super-resolution image and the one or more high resolution images of the at least the portion of the specimen to form a single coherent image of the at least the portion of the specimen as part of further processing the super-resolution image;
identifying a total number of artifacts in the single coherent image of the at least the portion of the specimen;
comparing the total number of artifacts in the single coherent image with a pre-defined tolerance number of artifacts to determine a relation of the total number of artifacts in the single coherent image with the pre-defined tolerance number of artifacts; and
further controlling operation of the super-resolution system to generate one or more high resolution images for the specimen based on one or more low resolution images of the specimen according to the relation of the total number of artifacts in the single coherent image with the pre-defined tolerance number of artifacts.

2. The method of claim 1 , wherein the one or more high resolution images of the at least the portion of the specimen obtained using the high resolution objective are high resolution images of one or more artifacts in the low resolution image.

3. The method of claim 1 , wherein the pre-defined tolerance number of artifacts is predefined based on one or more of a tolerance number of artifacts associated with the one or more related specimens, industry guidelines, hardware constraints, firmware constraints, software constraints, and constraints of an operator of the microscopy inspection system.

4. The method of claim 1 , further comprising:
detecting one or more artifacts in the low resolution image;
identifying a suitability of the one or more artifacts for generation of the super-resolution image of the least the portion of the specimen from the low resolution image; and
determining whether to further process the super-resolution image based on the suitability of the one or more artifacts for generation of the super-resolution image.

5. The method of claim 4 , wherein the one or more artifacts are identified as unsuitable for generating the super-resolution image.

6. The method of claim 5 , further comprising:
obtaining the one or more high resolution images of the one or more artifacts using the high resolution objective of the microscopy inspection system, if it is determined to further process the super-resolution image based on the suitability of the one or more artifacts for generation of the super-resolution image; and
assembling the super-resolution image and the one or more high resolution images of the one or more artifacts to form a single coherent image of the at least the portion of the specimen as part of further processing the super-resolution image.

7. The method of claim 4 , wherein the suitability of the one or more artifacts for generation of the super-resolution image is identified by a suitability classifier trained from a first group of known artifacts suitable for super-resolution imaging and a second group of known artifacts unsuitable for super-resolution imaging.

8. The method of claim 1 , wherein the simulated image classifier is trained using a plurality of known super-resolution images at two or more different image confidence determinations.

9. A super-resolution system comprising:
a microscopy inspection system for inspecting a specimen comprising:
a low resolution objective;
a high resolution objective;
one or more processors; and
at least one non-transitory computer-readable storage medium having stored therein instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
obtaining a low resolution image of the specimen using the low resolution objective of a microscopy inspection system;
generating a super-resolution image of at least a portion of the specimen from the low resolution image of the specimen using a super-resolution image simulation;
identifying an accuracy assessment of the super-resolution image based on one or more degrees of equivalence between the super-resolution image and one or more actually scanned high resolution images of at least a portion of one or more related specimens;
determining whether to further process the super-resolution image based on the accuracy assessment of the super-resolution image;
obtaining one or more high resolution images of the at least the portion of the specimen using a high resolution objective of the microscopy inspection system, if it is determined to further process the super-resolution image based on the accuracy assessment of the super-resolution image;
assembling the super-resolution image and the one or more high resolution images of the at least the portion of the specimen to form a single coherent image of the at least the portion of the specimen as part of further processing the super-resolution image;
identifying a total number of artifacts in the single coherent image of the at least the portion of the specimen;
comparing the total number of artifacts in the single coherent image with a pre-defined tolerance number of artifacts to determine a relation of the total number of artifacts in the single coherent image with the pre-defined tolerance number of artifacts; and
further controlling operation of the super-resolution system to generate one or more high resolution images for the specimen based on one or more low resolution images of the specimen according to the relation of the total number of artifacts in the single coherent image with the pre-defined tolerance number of artifacts.

10. The system of claim 9 , wherein the one or more high resolution images of the at least the portion of the specimen obtained using the high resolution objective are high resolution images of one or more artifacts in the low resolution image.

11. The system of claim 9 , wherein the instructions which, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:
detecting one or more artifacts in the low resolution image;
identifying a suitability of the one or more artifacts for generation of the super-resolution image of the least the portion of the specimen from the low resolution image; and
determining whether to further process the super-resolution image based on the suitability of the one or more artifacts for generation of the super-resolution image.

12. The system of claim 11 , wherein the one or more artifacts are identified as

Assignees

• Nanotronics Imaging Inc

Inventors

• Putman Matthew C
• Putman John B
• Pinskiy Vadim
• Succar Joseph R

Classifications

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title