Deep active learning method for civil infrastructure defect detection

Patent metadata

Field	Value
Publication number	US-10417524-B2
Application number	US-201715646563-A
Country	US
Kind code	B2
Filing date	Jul 11, 2017
Priority date	Feb 16, 2017
Publication date	Sep 17, 2019
Grant date	Sep 17, 2019

Abstract

An image processing system includes a memory to store a classifier and a set of labeled images for training the classifier, wherein each labeled image is labeled as either a positive image that includes an object of a specific type or a negative image that does not include the object of the specific type, wherein the set of labeled images has a first ratio of the positive images to the negative images. The system includes an input interface to receive a set of input images, a processor to determine a second ratio of the positive images, to classify the input images into positive and negative images to produce a set of classified images, and to select a subset of the classified images having the second ratio of the positive images to the negative images, and an output interface to render the subset of the input images for labeling.

First claim

We claim:

1. An image processing system, comprising:
a memory to store a classifier and a set of labeled images for training the classifier, each labeled image is labeled as either a positive image that includes an object of a specific type or a negative image that does not include the object of the specific type, wherein the set of labeled images has a first ratio of the positive images to the negative images;
an input interface to receive a set of input images;
a processor to determine, based on the first ratio, a second ratio of the positive images to the negative images, to classify, using the classifier, the input images into positive and negative images to produce a set of classified images, and to select a subset of the classified images having the second ratio of the positive images to the negative images, wherein the processor determines an uncertainty measure for each classified image indicating a confidence of the classifier that the classified image is the positive image or the negative image, orders the classified images according an order of their uncertainty measures into a sequence of positive images and a sequence of negative images, and selects a predetermined number of consecutive images from the sequence of positive images and a predetermined number of consecutive images from the sequence of negative images to satisfy the second ratio; and
an output interface to render the subset of the input images for labeling.

2. The image processing system of claim 1 , wherein the output interface is a display device having an interface allowing an operator to label a displayed image as the positive image or the negative image, wherein the processor renders the subset of the classified images on the display device, adds, in response to labeling the subset of classified images by the operator, the subset of the labeled classified images into the set of labeled images thereby updating the first ratio, and retrains the classifier with the updated set of labeled images.

3. The image processing system of claim 2 , wherein the processor trains the classifier using a weighted loss function penalizing incorrect positive and negative classification differently, wherein the weights of the weighted loss function are based on at least one or a combination of the first ratio and the second ratio.

4. The image processing system of claim 1 , wherein the object of the specific type is one or combination of a deposit of a surface, a crack of the surface, and effects of a water leakage of the surface.

5. The image processing system of claim 1 , wherein the labeled images and the input images include one or combination of intensity images and depth images.

6. The image processing system of claim 1 , wherein the classifier is a binary classifier.

7. The image processing system of claim 1 , wherein the classifier is a multi-label classifier, wherein the images are classified independently for each label.

8. The image processing system of claim 1 , wherein the classifier is a convolutional neural network.

9. The image processing system of claim 8 , wherein the convolutional neural network is a residual neural network.

10. The image processing system of claim 1 , wherein the second ratio is an inverse of the first ratio.

11. The image processing system of claim 1 , wherein the second ratio equals one when the first ratio is less than a threshold.

12. The image processing system of claim 1 , wherein the second ratio equals zero when the first ratio is greater than a threshold, and wherein an inverse of the second ratio equals zero when an inverse of the first ratio is less than a threshold.

13. A method for image processing, wherein the method uses a processor coupled to a memory storing a classifier and a set of labeled images for training the classifier, each labeled image is labeled as either a positive image that includes an object of a specific type or a negative image that does not include the object of the specific type, wherein the set of labeled images has a first ratio of the positive images to the negative images, wherein the processor is coupled with stored instructions implementing the method, wherein the instructions, when executed by the processor carry out at least some steps of the method, comprising:
receiving a set of input images;
determining, based on the first ratio, a second ratio of the positive images to the negative images;
determining an uncertainty measure for each classified image indicating a confidence of the classifier that the classified image is the positive image or the negative image;
ordering the classified images according an order of their uncertainty measures into a sequence of positive images and a sequence of negative images; and
selecting a predetermined number of consecutive images from the sequence of positive images and a predetermined number of consecutive images from the sequence of negative images to satisfy the second ratio;
classifying, using the classifier, the input images into positive and negative images to produce a set of classified images;
selecting a subset of the classified images having the second ratio of the positive images to the negative images; and
rendering the subset of the input images for labeling.

14. The method of claim 13 , further comprising:
adding, in response to labeling the subset of classified images, the subset of the labeled classified images into the set of labeled images thereby updating the first ratio; and
retraining the classifier with the updated set of labeled images.

15. The method of claim 14 , further comprising:
training the classifier using a weighted loss function penalizing incorrect positive and negative classification differently, wherein the weights of the weighted loss function are based on at least one or a combination of the first ratio and the second ratio.

16. The method of claim 13 , wherein the second ratio is an inverse of the first ratio.

17. The method of claim 13 , wherein the second ratio equals zero when the first ratio is greater than a threshold, and wherein an inverse of the second ratio equals zero when an inverse of the first ratio is less than a threshold.

18. A non-transitory computer readable storage medium storing a classifier and a set of labeled images for training the classifier, each labeled image is labeled as either a positive image that includes an object of a specific type or a negative image that does not include the object of the specific type, wherein the set of labeled images has a first ratio of the positive images to the negative image and embodied thereon a program executable by a processor for performing a method, the method comprising:
receiving a set of input images;
determining, based on the first ratio, a second ratio of the positive images to the negative images;
determining an uncertainty measure for each classified image indicating a confidence of the classifier that the classified image is the positive image or the negative image;
ordering the classified images according an order of their uncertainty measures into a sequence of positive images and a sequence of negative images; and
selecting a predetermined number of consecutive images from the sequence of positive images and a predetermined number of consecutive images from the sequence of negative images to satisfy the second ratio;
classifying, using the classifier, the input images into positive and negative images to produce a set of classified images;

selecting a subset of the classified images having the second ratio of the positive images to the nega

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Feng Chen
• Liu Ming-Yu
• Kao Chieh-Chi
• Lee Teng-Yok

Classifications

• G06V10/7753

  Incorporation of unlabelled data, e.g. multiple instance learning [MIL] · CPC title

• G06V10/7784

  based on feedback from supervisors · CPC title

• G06V10/772

  Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title