Semi-supervised learning based on clustering objects in video from a property

Patent metadata

Field	Value
Publication number	US-11544505-B1
Application number	US-202016829402-A
Country	US
Kind code	B1
Filing date	Mar 25, 2020
Priority date	Apr 1, 2019
Publication date	Jan 3, 2023
Grant date	Jan 3, 2023

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for semi-supervised training of an object recognition model. The methods, systems, and apparatus include a monitoring system including a camera located at a property and configured to generate images and one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform actions of determining a cluster of images meets a threshold for number of included images and a threshold for cluster tightness. A representative image of the cluster is selected and a query including the representative image of the cluster is provided. User feedback responsive to the query is received and an object recognition model is updated based on the user feedback.

First claim

What is claimed is:

1. A monitoring system that is configured to monitor a property, the monitoring system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
determining that a cluster of images meets a threshold for a number of included images and a threshold for cluster tightness;
selecting a representative image of the cluster;
providing a query including the representative image of the cluster;
receiving user feedback responsive to the query; and
updating an object recognition model based on the user feedback, the updating comprising:
determining that the user feedback is negative; and
in response to determining that the user feedback is negative, combining the cluster of images and another cluster of images.

2. The system of claim 1 , wherein updating an object recognition model based on the user feedback comprises:
determining that the user feedback is negative; and
in response to determining that the user feedback is negative, retraining the cluster of images to include a different set of images.

3. The system of claim 1 , wherein updating an object recognition model based on the user feedback comprises:
determining that the user feedback is positive; and
in response to determining that the user feedback is positive, updating the object recognition model with the cluster of images.

4. The system of claim 1 , wherein updating an object recognition model based on the user feedback comprises:
determining that the user feedback is positive; and
in response to determining that the user feedback is positive, reinforcing the cluster.

5. The system of claim 1 , wherein selecting a representative image comprises:
selecting a particular image from a center of the cluster,
wherein the query requests that a user identify an object in the particular image.

6. The system of claim 1 , wherein selecting a representative image comprises:
selecting a particular image from an edge of the cluster,
wherein the query provides an identity of an object in the particular image and requests that a user verify the identity of the object.

7. The system of claim 1 , the operations comprising:
detecting an object based on the object recognition model; and
providing, to a user, an alert that indicates that the object was detected.

8. The monitoring system of claim 1 , wherein:
the cluster of images comprises a first plurality of extracted features with respective weights;
the other cluster of images comprises a second plurality of extracted features with respective weights; and
combining the cluster of images and the other cluster of images comprises adjusting the respective weights of the first plurality of extracted features and the respective weights of the second plurality of extracted features.

9. The monitoring system of claim 8 , wherein adjusting the respective weights of the first plurality of extracted features and the respective weights of the second plurality of extracted features reduces a distance between the cluster and the other cluster of images.

10. A computer-implemented method comprising:
determining that a cluster of images meets a threshold for a number of included images and a threshold for cluster tightness;
selecting a representative image of the cluster;
providing a query including the representative image of the cluster;
receiving user feedback responsive to the query; and
updating an object recognition model based on the user feedback, the updating comprising:
determining that the user feedback is negative; and
in response to determining that the user feedback is negative, combining the cluster of images and another cluster of images.

11. The method of claim 10 , wherein updating an object recognition model based on the user feedback comprises:
determining that the user feedback is negative; and
in response to determining that the user feedback is negative, retraining the cluster of images to include a different set of images.

12. The method of claim 10 , wherein updating an object recognition model based on the user feedback comprises:
determining that the user feedback is positive; and
in response to determining that the user feedback is positive, updating the object recognition model with the cluster of images.

13. The method of claim 10 , wherein updating an object recognition model based on the user feedback comprises:
determining that the user feedback is positive; and
in response to determining that the user feedback is positive, reinforcing the cluster.

14. The method of claim 10 , wherein selecting a representative image comprises:
selecting a particular image from a center of the cluster,
wherein the query requests that a user identify an object in the particular image.

15. The method of claim 10 , wherein selecting a representative image comprises:
selecting a particular image from an edge of the cluster,
wherein the query provides an identity of an object in the particular image and requests that a user verify the identity of the object.

16. The method of claim 10 , comprising:
detecting an object based on the object recognition model; and
providing, to a user, an alert that indicates that the object was detected.

17. The method of claim 10 , wherein:
the cluster of images comprises a first plurality of extracted features with respective weights;
the other cluster of images comprises a second plurality of extracted features with respective weights; and
combining the cluster of images and the other cluster of images comprises adjusting the respective weights of the first plurality of extracted features and the respective weights of the second plurality of extracted features.

18. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
determining that a cluster of images meets a threshold for a number of included images and a threshold for cluster tightness;
selecting a representative image of the cluster;
providing a query including the representative image of the cluster;
receiving user feedback responsive to the query; and
updating an object recognition model based on the user feedback, the updating comprising:
determining that the user feedback is negative; and
in response to determining that the user feedback is negative, combining the cluster of images and another cluster of images.

19. The computer-readable medium of claim 18 , wherein updating an object recognition model based on the user feedback comprises:
determining that the user feedback is negative; and
in response to determining that the user feedback is negative, retraining the cluster of images to include a different set of images.

20. The computer-readable medium of claim 18 , wherein updating an object recognition model based on the user feedback comprises:
determining that the user feedback is positive; and
in response to determining that the user feedback is positive, updating the object recognition model with the cluster of images.

Assignees

• Alarm Com Inc

Inventors

• Beach Allison
• Berg Benjamin Asher
• Lee Sung Chun
• Taheri Sima
• Bondugula Sravanthi
• Salmani Niyasar Nafiseh
• Qian Gang

Classifications

• G06F18/2155 Primary

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title

• G06F18/2178

  based on feedback of a supervisor · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06F18/23211

  with adaptive number of clusters · CPC title

• G06K9/6263

  Physics · mapped topic