Processing cell images using neural networks

Patent metadata

Field	Value
Publication number	US-11443190-B2
Application number	US-202016905714-A
Country	US
Kind code	B2
Filing date	Jun 18, 2020
Priority date	Feb 26, 2016
Publication date	Sep 13, 2022
Grant date	Sep 13, 2022

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing cell images using neural networks. One of the methods includes obtaining data comprising an input image of one or more biological cells illuminated with an optical microscopy technique; processing the data using a stained cell neural network; and processing the one or more stained cell images using a cell characteristic neural network, wherein the cell characteristic neural network has been configured through training to receive the one or more stained cell images and to process the one or more stained cell images to generate a cell characteristic output that characterizes features of the biological cells that are stained in the one or more stained cell images.

First claim

The invention claimed is:

1. An apparatus comprising:
one or more storage devices for storing data representing a convolutional neural network, and
one or more processors communicatively coupled to the one or more storage devices and configured to perform operations associated with the convolutional neural network, the operations comprising:
obtaining input data comprising one or more images of one or more biological cells, wherein each image is an image of the one or more biological cells illuminated with a respective microscopy technique of one or more microscopy techniques;
processing the input data comprising the one or more images using the convolutional neural network, wherein the convolutional neural network has been configured through training
to receive the input data comprising the one or more images of the one or more biological cells, and
to process respective pixels of the one or more images of the input data to perform fully convolutional image-to-image regression from the input data to predict one or more fluorescent channels of an output image of the one or more biological cells,
wherein each pixel in each of the one or more fluorescent channels generated by the convolutional neural network corresponds to a respective pixel of the one or more images; and

providing the output image with the one or more fluorescent channels.

2. The apparatus of claim 1 , wherein the apparatus is operatively coupled to a microscopic camera, and wherein the operations further comprise generating the one or more images of the one or more biological cells using the microscopic camera.

3. The apparatus of claim 2 , wherein the generating the one or more images of the one or more biological cells comprises generating the one or more images to have different focal points.

4. The apparatus of claim 1 , wherein the one or more microscopy techniques include one or more of an optical microscopy technique, a hyperspectral microscopy technique, or a fluorescent microscopy technique.

5. The apparatus of claim 1 , wherein the apparatus is operatively coupled to a display device, and wherein providing the output image comprises providing the output image for display on the display device.

6. The apparatus of claim 1 , wherein the operations further comprise:
processing the output image using a cell characteristic neural network, wherein the cell characteristic neural network has been trained to generate a cell characteristic output that characterizes features of the one or more biological cells in the output image.

7. The apparatus of claim 6 , wherein the cell characteristic output comprises data characterizing one or more of a cell type, a cell health, a cell population, a cell phenotype, or a cell state of differentiation in induced pluripotent stem cell protocols of the one or more biological cells.

8. The apparatus of claim 6 , wherein the cell characteristic output specifies, from the one or more biological cells, an average cell health or a percentage of cells that have correctly differentiated through induced pluripotent stem cell protocols.

9. The apparatus of claim 6 , wherein the cell characteristic output specifies a cell condition for each biological cell of the one or more biological cells.

10. A method comprising,
training a convolutional neural network comprising one or more parameter values and that is configured to receive input data comprising one or more images of one or more biological cells, and to process respective pixels of the one or more images to perform fully convolutional image-to-image regression from the one or more images of the input data to predict one or more fluorescent channels of an output image of the one or more biological cells,
wherein the training comprises:
obtaining training data comprising one or more images of one or more biological cells and one or more stained images depicting the one or more biological cells, each stained image depicting the one or more biological cells according to a respective fluorescent channel of one or more fluorescent channels;
processing the one or more images through the convolutional neural network to obtain one or more fluorescent channels of a predicted output image;
determining an error between the one or more stained images and the one or more fluorescent channels of the predicted output image, comprising comparing a respective error between each pixel of each fluorescent channel of the predicted output image, with each corresponding pixel in a respective stained image corresponding to the fluorescent channel; and
updating the one or more parameter values of the convolutional neural network according to the determined error.

11. The method of claim 10 , wherein the one or more images of the training data are each of the one or more biological cells illuminated with a respective microscopy technique of one or more microscopy techniques.

12. The method of claim 11 , wherein the one or more microscopy techniques include one or more of an optical microscopy technique, a hyperspectral microscopy technique, or a fluorescent microscopy technique.

13. The method of claim 11 , wherein the one or more images of the training data have different focal points.

14. The method of claim 10 , further comprising:
processing input data comprising one or more second images of one or more second biological cells to predict, through the convolutional neural network, one or more fluorescent channels of an output image of the one or more second biological cells.

15. The method of claim 14 , further comprising:
processing the output image of the one or more second biological cells using a cell characteristic neural network, wherein the cell characteristic neural network has been configured through training to generate a cell characteristic output that characterizes features of the one or more second biological cells in the output image.

16. The method of claim 15 , wherein the cell characteristic output comprises data characterizing one or more of a cell type, a cell health, a cell population, a cell phenotype, or a cell state of differentiation in induced pluripotent stem cell protocols of the one or more second biological cells.

17. The method of claim 15 , wherein the cell characteristic output specifies, from the one or more second biological cells, an average cell health or a percentage of cells that have correctly differentiated through induced pluripotent stem cell protocols.

18. The method of claim 15 , wherein the cell characteristic output specifies a cell condition for each biological cell of the one or more second biological cells.

19. A system comprising:
one or more one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
training a convolutional neural network comprising one or more parameter values and that is configured to receive input data comprising one or more images of one or more biological cells, and to process respective pixels of the one or more images to perform fully convolutional image-to-image regression from the one or more images of the input data to predict one or more fluorescent channels of an output image of the one or more biological cells,
wherein the training comprises:
obtaining training data comprising one or more images of one or more biological cells and one or more stained images depicting the one or more biological cells, each stained image depicting the one or more biological cells according to a respective

Assignees

• Google Llc

Inventors

• Nelson Philip Charles
• Christiansen Eric Martin
• Berndl Marc
• Frumkin Michael

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06T7/0014 Primary

  using an image reference approach · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/02

  Neural networks · CPC title