System and method for adaptive, rapidly deployable, human-intelligent sensor feeds

Patent metadata

Field	Value
Publication number	US-10657385-B2
Application number	US-201615561407-A
Country	US
Kind code	B2
Filing date	Mar 25, 2016
Priority date	Mar 25, 2015
Publication date	May 19, 2020
Grant date	May 19, 2020

Abstract

The disclosure describes a sensor system that provides end users with intelligent sensing capabilities, and embodies both crowd sourcing and machine learning together. Further, a sporadic crowd assessment is used to ensure continued sensor accuracy when the system is relying on machine learning analysis. This sensor approach requires minimal and non-permanent sensor installation by utilizing any device with a camera as a sensor host, and provides human-centered and actionable sensor output.

First claim

What is claimed is:

1. An adaptive sensor system comprising:
a sensor, wherein the sensor provides a sensor feed of an environment;
a user interface,
wherein a user can enter a question to be answered by the sensor system using the user interface,
wherein the question relates to a condition of the environment;

a dispatcher, wherein the dispatcher directs the sensor feed and question to at least one of a crowd module and a machine learning module,
wherein at least one of the crowd module and the machine learning module generates a labeled sensor feed derived from the sensor feed,
wherein the labeled sensor feed comprises an answer to the question; and

a periodic comparator that evaluates a relative accuracy of the labeled sensor feed generated by the machine learning module as compared to the labeled sensor feed generated by the crowd module.

2. The adaptive sensor system of claim 1 , wherein the dispatcher directs the sensor feed to the crowd module upon initiation of the sensor system and prior to the machine learning module reaching a threshold level of accuracy.

3. The adaptive sensor system of claim 2 ,
wherein the labeled sensor feeds generated by the crowd module are sent to the machine learning module,
wherein the machine learning module uses the labeled sensor feeds to train a classifier.

4. The adaptive sensor system of claim 3 , wherein the dispatcher directs the sensor feed to the machine learning module when the machine learning module is operating above a threshold level of accuracy.

5. The adaptive sensor system of claim 1 , wherein the crowd module sends the sensor feed and question to crowd workers to generate the labeled sensor feed.

6. The adaptive sensor system of claim 1 , wherein the user interface provides a visualization of the labeled sensor feed.

7. The adaptive sensor system of claim 1 , wherein the sensor comprises an electronic device having a camera.

8. The adaptive sensor system of claim 1 , wherein the sensor feed comprises an image.

9. The adaptive sensor system of claim 1 , wherein the question has a data type selected from the group consisting of: yes/no, number, scale, and multiple choice.

10. The adaptive sensor system of claim 1 , further comprising :
an image similarity detector that receives the sensor feed, wherein the image similarity detector maintains a current sensor value if there is no change in the condition of the environment over a period of time.

11. A method of creating an adaptive sensor system, comprising:
providing a sensor that provides a sensor feed;
receiving from a user interface a question, wherein the question is capable of being
answered by observation of the sensor feed;

sending the sensor feed to a crowd,
wherein the crowd generates a labeled sensor feed based on the sensor feed;

using the crowd labeled sensor feed to train a machine learning module;
sending a subsequent sensor feed to the machine learning module to generate the labeled sensor feed when the machine learning module exceeds a threshold level of accuracy; and
sending the labeled sensor feed to a user via the user interface wherein the labeled sensor feed comprises an answer to the question.

12. The method of claim 11 , further comprising:
periodically testing the labeled sensor feed generated by the machine learning module against the labeled sensor feed generated by the crowd to determine the relative accuracy of the machine learning module.

13. The method of claim 12 , wherein the sensor feed is sent to the crowd if the machine learning module falls below the threshold level of accuracy.

14. The method of claim 11 , further comprising:
choosing a region of interest in the sensor feed prior to sending the sensor feed to the crowd.

15. The method of claim 11 , further comprising:
obfuscating the sensor feed.

16. The method of claim 11 , further comprising:
rejecting a second sensor feed received subsequent to a first sensor feed if there is no difference between the first sensor feed and the second sensor feed.

17. The method of claim 16 , wherein the second sensor feed is rejected if a pixel difference threshold does not exceed 10%.

18. The method of claim 16 , wherein the second sensor feed is rejected if an image area percentage threshold does not exceed 1%.

19. The method of claim 11 , wherein sending the sensor feed to a crowd to generate a labeled sensor feed further comprises:
sending an example sensor feed to the crowd.

Assignees

• Carnegie Mellon Univ A Pennsylvania Non Profit Corporation
• Univ Rochester
• Carnegie Mellon Univ A Pennsylvania Non Pro Fit Corporation

Inventors

• Laput Gierad
• Harrison Christopher
• Bigham Jeffrey P
• Lasecki Walter S
• Xiao Bo Robert
• Wiese Jason

Classifications

• G06N5/041

  Abduction · CPC title

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F3/01

  Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) · CPC title

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title