Automated control of microfluidic devices based on machine learning

Patent metadata

Field	Value
Publication number	US-2021302452-A1
Application number	US-202117344183-A
Country	US
Kind code	A1
Filing date	Jun 10, 2021
Priority date	Apr 26, 2017
Publication date	Sep 30, 2021
Grant date	—

Abstract

A system is provided to automatically monitor and control the operation of a microfluidic device using machine learning technology. The system receives images of a channel of a microfluidic device collected by a camera during operation of the microfluidic device. Upon receiving an image, the system applies a classifier to the image to classify the operation of the microfluidic device as normal, in which no adjustment to the operation is needed, or as abnormal, in which an adjustment to the operation is needed. When an image is classified as normal, the system may make no adjustment to the microfluidic device. If, however, an image is classified as abnormal, the system may output an indication that the operation is abnormal, output an indication of a needed adjustment, or control the microfluidic device to make the needed adjustment.

First claim

1 - 45 . (canceled)

46 . A method performed by a computing system, the method comprising:
collecting images relating to operation of a training microfluidic device during operation of the training microfluidic device; generating a label for each image relating to operation of the training microfluidic device; and training a classifier using the images and labels as training data, the classifier for receiving an image relating to operation of a production microfluidic device and generating a label relating to operation of the production microfluidic device.

47 . The method of claim 46 wherein the classifier includes a convolutional neural network and a sub-classifier.

48 . The method of claim 47 wherein the sub-classifier is a support vector machine.

49 . The method of claim 47 wherein the sub-classifier is a linear regression classifier.

50 . The method of claim 47 wherein the convolutional neural network inputs and generates a feature vector for the image and wherein the sub-classifier inputs the feature vector and generates a label for the image.

51 . The method of claim 47 wherein the convolutional neural network include multiple convolutional layers, wherein each convolutional layer is associated with a rectified linear unit and a max pooling technique, wherein the input to the first convolutional layer is an image, wherein the output of each convolutional layer is processed by the associated rectified linear unit and max pooling technique to generate output that is input to the next convolutional layer if any and input to a fully connected layer if there is no next convolutional layer.

52 . The method of claim 47 wherein the sub-classifier generates the label for an image further based on an additional feature not generated by the convolution neural network.

53 . The method of claim 46 wherein a label indicates an abnormal operation of the training microfluidic device.

54 . The method of claim 46 wherein the microfluidic device includes a micromixer for mixing two or more fluids with a microchannel.

55 . The method of claim 46 wherein the microfluidic device includes a microswitch and the labels indicate shapes of cells.

56 . The method of claim 46 wherein the microfluidic device is a microencapsulation device and a label indicates whether an image is of a well-formed microcapsule or a malformed microcapsules.

57 - 73 . (canceled)

74 . One or more computing systems for training a classifier for generating a label for an image by applying a classifier, the one or more computing systems comprising:
one or more computer-readable storage mediums for storing computer-executable instructions for controlling the one or more computing systems to:
access training data for training the classifier, the training data including images having labels the images and labels relating to operation of a fluidic device;
training the classifier using the training data
wherein the classifier includes a convolutional neural network and a sub-classifier and wherein during classification, an image is input to the convolutional neural network, output of the convolutional neural network is input to the sub-classifier as a feature vector for the image, and output of the sub-classifier is a label for the image; and
one or more processors for executing the computer-executable instructions stored in the one or more computer-readable storage mediums.

75 . The one or more computing systems claim 74 wherein the sub-classifier is a support vector machine.

76 . The one or more computing systems of claim 74 wherein the sub-classifier is a linear regression classifier.

77 . The one or more computing systems of claim 74 wherein the convolutional neural network includes multiple convolutional layers, wherein each convolutional layer is associated with a rectified linear unit and a max pooling technique, wherein the input to the first convolutional layer is the image, wherein the output of each convolutional layer is processed by the associated rectified linear unit and max pooling technique to generate output that is input to the next convolutional layer if any and input to a fully connected layer if there is no next convolutional layer.

78 . The one or more computing systems of claim 74 wherein the sub-classifier inputs an additional feature not generated by the convolutional neural network.

79 . The method of claim 46 wherein the microfluidic device includes a micromixer for mixing two or more fluids with a microchannel.

80 . The method of claim 79 wherein the two or more fluids are reactants.

81 . The method of claim 79 wherein the two or more fluids include an acid or a base to control pH balance in another fluid.

82 . The method of claim 46 wherein a label relates to a change in a flow rate of a fluid.

83 . The computing system of claim 46 wherein a label relates to discarding malformed droplets of a double emulsion.

84 . The method of claim 46 wherein during initial operation of the microfluidic device, a label relates to controlling the microfluidic device to operate normally.

85 . The method of claim 46 wherein the microfluidic device includes a switch and wherein a label relates to controlling the switch to sort cells of a target shape into a target bin and other cells into another bin.

86 . The method of claim 46 wherein the microfluidic device is a microencapsulation device with a switch and wherein a label relates to controlling the switch to deliver well-formed microcapsules to one bin and malformed microcapsules to another bin.

87 . The one or more computing systems of claim 74 wherein a label relates to controlling the fluidic device.

88 . The one or more computing systems of claim 74 wherein a label relates to a condition of operation of the fluidic device

89 . The one or more computing systems of claim 74 wherein the classifier includes a convolution neural network to generate features for the images and a sub-classifier to generate labels for the images based on the generated features of the images.

90 . The one or more computing systems of claim 74 wherein the sub-classifier generates the labels for the images further based on an additional feature not generated by the convolution neural network.

91 . One or more computing systems comprising:
one or more computer-readable storage mediums for storing
images relating to operation of a fluidic device, each image having a label relating to operation of the fluidic device; and
computer-executable instructions for controlling the one or more computing systems to train a classifier using the images and labels as training data, the classifier for receiving an image relating to operation of a production fluidic device and for generating a label relating to operation of the production fluidic device; and
one or more processors for executing the computer-executable instructions stored in the one or more computer-readable storage mediums.

92 . The one or more computing systems of claim 91 wherein a label indicates an abnormal operation of a fluidic device.

93 . The one or more computing systems of claim 91 wherein when a label indicates that flow rate of a fluid that enters a fluidic device is to be adjusted.

94 . The one or more c

Assignees

• L Livermore Nat Security Llc

Inventors

• Giera Brian
• Duoss Eric B
• Nguyen Du
• Smith William
• Talathi Sachin Subhash
• Wilson Aaron Creighton
• Ye Congwang

Classifications

• G05D7/0694 Primary

  by action on throttling means or flow sources of very small size, e.g. microfluidics (microvalves F16K99/0001; microstructural devices per se B81B) · CPC title

• G01N35/00871 Primary

  Communications between instruments or with remote terminals · CPC title

• B01L3/502784

  specially adapted for droplet or plug flow, e.g. digital microfluidics · CPC title

• G05B2219/37333

  Position of control valve and position of controlled actuator · CPC title

• B01L2200/143

  Quality control, feedback systems · CPC title