Method and system for mapping objects on unknown specimens

Patent metadata

Field	Value
Publication number	US-11815673-B2
Application number	US-202217663599-A
Country	US
Kind code	B2
Filing date	May 16, 2022
Priority date	Oct 19, 2018
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

A method and system for mapping fluid objects on a substrate using a microscope inspection system that includes a light source, imaging device, stage for moving a substrate disposed on the stage, and a control module. A computer analysis system includes an object identification module that identifies for each of the objects on the substrate, an object position on the substrate including a set of X, Y, and θ coordinates using algorithms, networks, machines and systems including artificial intelligence and image processing algorithms. At least one of the objects is fluid and has shifted from a prior position or deformed from a prior size.

First claim

The invention claimed is:

1. A method comprising:
receiving, by a computing system, a scan of a substrate using a microscope inspection system, the substrate comprising one or more objects;
for each object on the substrate, classifying, by an artificial intelligence model of the computing system, a type corresponding to the object;
for each object on the substrate, identifying, by the computing system, an initial object position on the substrate; and
predicting, by the artificial intelligence model, a future position of each object on the substrate and an anticipated amount of deformity for each object at the future position based on the type of object and the initial object position.

2. The method of claim 1 , further comprising:
training the artificial intelligence model to classify types of objects by:
generating a training data set comprising a plurality of labeled images, wherein the plurality of labeled images comprises deformed objects; and
causing the artificial intelligence model to learn a classification of each deformed object based on the training data set.

3. The method of claim 1 , further comprising:
training the artificial intelligence model to classify types of objects by:
generating a training data set comprising a plurality of labeled images, wherein the plurality of labeled images comprises rotated objects; and
causing the artificial intelligence model to learn a classification of each rotated object based on the training data set.

4. The method of claim 1 , wherein, for each object on the substrate, identifying, by the computing system, the initial object position on the substrate comprises:
generating an initial object layout map for the substrate, wherein the initial object layout map comprises the initial object position of each object.

5. The method of claim 4 , wherein predicting, by the artificial intelligence model, the future position of each object on the substrate comprises:
predicting the future position of each object based on the initial object layout map for the substrate.

6. The method of claim 1 , further comprising:
generating an alert upon determining that the future position of each object exceeds a threshold tolerance from the initial object position.

7. A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by a processor, causes a computing system to perform operations comprising:
receiving, by the computing system, a scan of a substrate using a microscope inspection system, the substrate comprising one or more objects;
for each object on the substrate, classifying, by an artificial intelligence model of the computing system, a type corresponding to the object;
for each object on the substrate, identifying, by the computing system, an initial object position on the substrate; and
predicting, by the artificial intelligence model, a future position of each object on the substrate and an anticipated amount of deformity for each object at the future position based on the type of object and the initial object position.

8. The non-transitory computer readable medium of claim 7 , further comprising:
training the artificial intelligence model to classify types of objects by:
generating a training data set comprising a plurality of labeled images, wherein the plurality of labeled images comprises deformed objects; and
causing the artificial intelligence model to learn a classification of each deformed object based on the training data set.

9. The non-transitory computer readable medium of claim 7 , further comprising:
training the artificial intelligence model to classify types of objects by:
generating a training data set comprising a plurality of labeled images, wherein the plurality of labeled images comprises rotated objects; and
causing the artificial intelligence model to learn a classification of each rotated object based on the training data set.

10. The non-transitory computer readable medium of claim 7 , wherein, for each object on the substrate, identifying, by the computing system, the initial object position on the substrate comprises:
generating an initial object layout map for the substrate, wherein the initial object layout map comprises the initial object position of each object.

11. The non-transitory computer readable medium of claim 10 , wherein predicting, by the artificial intelligence model, the future position of each object on the substrate comprises:
predicting the future position of each object based on the initial object layout map for the substrate.

12. The non-transitory computer readable medium of claim 7 , further comprising:
generating an alert upon determining that the future position of each object exceeds a threshold tolerance from the initial object position.

13. A system comprising:
a processor; and
a memory having programming instructions stored thereon, which, when executed by the processor, causes the system to perform operations comprising:
receiving a scan of a substrate using a microscope inspection system, the substrate comprising one or more objects;
for each object on the substrate, classifying, by an artificial intelligence model, a type corresponding to the object;
for each object on the substrate, identifying an initial object position on the substrate; and
predicting, by the artificial intelligence model, a future position of each object on the substrate and an anticipated amount of deformity for each object at the future position based on the type of object and the initial object position.

14. The system of claim 13 , wherein the operations further comprise:
training the artificial intelligence model to classify types of objects by:
generating a training data set comprising a plurality of labeled images, wherein the plurality of labeled images comprises deformed objects; and
causing the artificial intelligence model to learn a classification of each deformed object based on the training data set.

15. The system of claim 13 , further comprising:
training the artificial intelligence model to classify types of objects by:
generating a training data set comprising a plurality of labeled images, wherein the plurality of labeled images comprises rotated objects; and
causing the artificial intelligence model to learn a classification of each rotated object based on the training data set.

16. The system of claim 13 , wherein, for each object on the substrate, identifying the initial object position on the substrate comprises:
generating an initial object layout map for the substrate, wherein the initial object layout map comprises the initial object position of each object.

17. The system of claim 16 , wherein predicting, by the artificial intelligence model, the future position of each object on the substrate comprises:
predicting the future position of each object based on the initial object layout map for the substrate.

Assignees

• Nanotronics Imaging Inc

Inventors

• Putman Matthew C
• Putman John B
• Cruickshank John
• Orlando Julie
• Frankel Adele
• Scott Brandon

Classifications

• G02B21/365 Primary

  Control or image processing arrangements for digital or video microscopes (G02B21/361, G02B21/362 take precedence) · CPC title

• G02B21/0032

  Optical details of illumination, e.g. light-sources, pinholes, beam splitters, slits, fibers (G02B21/0036 - G02B21/008; means for illumination of specimens in general G02B21/06) · CPC title

• G02B21/361

  Optical details, e.g. image relay to the camera or image sensor (G02B21/364 takes precedence; illumination details G02B21/06 and subgroups) · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06V20/69

  Microscopic objects, e.g. biological cells or cellular parts · CPC title