Computer vision system for industrial equipment gauge digitization and alarms

Patent metadata

Field	Value
Publication number	US-11443133-B2
Application number	US-201916709035-A
Country	US
Kind code	B2
Filing date	Dec 10, 2019
Priority date	Dec 10, 2019
Publication date	Sep 13, 2022
Grant date	Sep 13, 2022

Abstract

A system for analog gauge monitoring uses a machine learning model for computer vision that is trained using synthetic training data generated based on one or a few images of the gauge being monitored and a geometric model describing the scale and the indicator of the gauge. In some embodiments, the synthetic training data is generated using an image model implemented as a generative adversarial network (GAN) type neural network and trained to modify an image of a given gauge such that the gauge face is preserved while the gauge indicator is added to or removed from the image of the given gauge for any given gauge.

First claim

What is claimed is:

1. A method for monitoring a gauge having a face including markings denoting a scale for a parameter value being measured by the gauge and an indicator indicating a reading of the parameter value over the scale by displacement of the indicator on the face, the method comprising:
receiving a set of training images of the gauge;
receiving data relating to annotations of the training images of the gauge, the annotations denoting the scale and the indicator on the face of the gauge;
generating a geometric model describing the gauge using geometric information derived from the data relating to the annotations;
providing an image model previously trained to modify an image of a given gauge such that a gauge face is preserved while a gauge indicator is added to or removed from the image of the given gauge for any given gauge, the previously trained image model being a neural network that is trained using a mask to distinguish the gauge indicator from the gauge face for any given gauge;
generating, using the geometric model and the previously trained image model, synthetic images of the gauge, the synthetic images having indicator at different positions relative to the scale, the synthetic images forming a training data set, wherein generating the synthetic images of the gauge comprises:
selecting an image patch from a respective training image of the gauge as input to the previously trained image model;
providing a reference image patch of the indicator of the gauge to the previously trained image model; and
applying the previously trained image model to erase the indicator from or draw the indicator onto the selected image patch;

training a machine learning model for computer vision using the training data set;
receiving operation images of the gauge during normal operation of the gauge; and
applying the trained machine learning model for computer vision to the operation images to predict readings of the gauge.

2. The method of claim 1 , wherein receiving the set of training images of the gauge comprises:
receiving one to five training images of the gauge as the set of training images, each training image being annotated to denote the scale and the indicator on the face of the gauge in that training image.

3. The method of claim 1 , wherein receiving data relating to annotations of the training images of the gauge comprises:
receiving data relating to the annotations denoting at least end points of the scale and at least end points of the indicator on the face of the gauge.

4. The method of claim 3 , further comprising:
receiving data relating to the annotations denoting a point on the scale between two endpoints.

5. The method of claim 1 , wherein generating a geometric model describing the gauge using geometric information derived from the data relating to the annotations comprises:
generating the geometric model describing a geometric location of the scale on the face of the gauge and a geometric location and movement of the indicator on the face of the gauge.

6. The method of claim 1 , wherein the previously trained image model is a generative adversarial network (GAN).

7. The method of claim 6 , wherein selecting the image patch from the respective training image of the gauge includes selecting the image patch from the respective training image of the gauge using the geometric model.

8. The method of claim 7 , wherein generating, using the geometric model and the previously trained image model, the synthetic images of the gauge further comprises:
selecting a first image patch from a respective training image of the gauge as input to the previously trained image model, the first image patch containing the indicator of the gauge;
applying the previously trained image model to erase the indicator from the first image patch;
pasting the modified first image patch back onto the respective training image to generate a base image of the gauge without the indicator;
rotating the base image in a first direction by a first number of degrees;
selecting a second image patch from the base image of the gauge as input to the previously trained image model;
applying the previously trained image model to draw the indicator onto the second image patch;
pasting the modified second image patch back onto the base image to generate a synthetic image;
rotating the synthetic image in a second direction opposite the first direction by the first number of degrees;
providing the synthetic image as part of the training data set; and
repeating the rotating the base image using a different values of rotation degrees to providing the synthetic image to generate a plurality of synthetic images forming the training data set.

9. The method of claim 1 , further comprising:
training the machine learning model for computer vision using training data denoting a threshold value for the gauge; and
applying the trained machine learning model for computer vision to predict readings from operation images of the gauge as being above or below the threshold value.

10. The method of claim 1 , further comprising:
determining a time duration after receiving a first operation image; and
in response to the time duration exceeding a first time value, generating an alarm indicating that no subsequent operation image is being received for the time duration exceeding the first time value.

11. A system for monitoring a gauge having a face including markings denoting a scale for a parameter value being measured by the gauge and an indicator indicating a reading of the parameter value over the scale by displacement of the indicator on the face, the system comprising:
a hardware processor; and
a memory coupled with the hardware processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to:
receive a set of training images of the gauge;
receive data relating to annotations of the training images of the gauge, the annotations denoting the scale and the indicator on the face of the gauge;
generate a geometric model describing the gauge using geometric information derived from the data relating to the annotations;
providing an image model previously trained to modify an image of a given gauge such that a gauge face is preserved while a gauge indicator is added to or removed from the image of the given gauge for any given gauge, the previously trained image model being a neural network that is trained using a mask to distinguish the gauge indicator from the gauge face for any given gauge;
generate, using the geometric model and the previously trained image model, synthetic images of the gauge, the synthetic images having indicator at different positions relative to the scale, the synthetic images forming a training data set, wherein to generate the synthetic images of the gauge, the instructions further cause the processor to:
select an image patch from a respective training image of the gauge as input to the previously trained image model;
provide a reference image patch of the indicator of the gauge to the previously trained image model; and
apply the previously trained image model to erase the indicator from or draw the indicator onto the selected image patch;

train a machine learning model for computer vision using the training data set;
receive operation images of the gauge during normal operation of the gauge; and
apply the trained machine learning model for computer vision to the operation images to predict readings of the gauge.

12. The system of claim 11 , wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to:

Assignees

• Schwartz Edward L
• Ricoh Co Ltd

Inventors

• Schwartz Edward L

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2113

  by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation · CPC title

• G06N3/088 Primary

  Non-supervised learning, e.g. competitive learning · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title