Activity classification based on multi-sensor input

Patent metadata

Field	Value
Publication number	US-11550276-B1
Application number	US-202016857271-A
Country	US
Kind code	B1
Filing date	Apr 24, 2020
Priority date	Apr 24, 2019
Publication date	Jan 10, 2023
Grant date	Jan 10, 2023

Abstract

A method for classifying activity based on multi-sensor input includes receiving, from two or more sensors, sensor data indicating activity within a building, determining, for each of the two or more sensors and based on the received sensor data, (i) an extracted feature vector for activity within the building and (ii) location data, labelling each of the extracted feature vectors with the location data, generating, using the extracted feature vectors, an integrated feature vector, detecting a particular activity based on the integrated feature vector, and in response to detecting the particular activity, performing a monitoring action.

First claim

What is claimed is:

1. A method comprising:
receiving, from two or more sensors, sensor data indicating activity within a building;
determining, for each of the two or more sensors and using the received sensor data, (i) an extracted feature vector for the activity within the building and (ii) location data;
labelling each of the extracted feature vectors with the location data;
generating, using the labelled extracted feature vectors, an integrated feature vector;
determining, using the integrated feature vector as input to an activity classifier, a classification of the activity within the building and a location of the activity; and
in response to determining the classification of the activity within the building and the location of the activity, performing a monitoring action.

2. The method of claim 1 , wherein the location data indicates a location of the sensor.

3. The method of claim 1 , wherein the integrated feature vector represents a composite of two or more different extracted feature vectors that each represent one or more features of a single sensor.

4. The method of claim 3 , wherein the one or more features of the single sensor include sensor specification data.

5. The method of claim 1 , wherein determining the classification of the activity within the building and the location of the activity comprises classifying the received sensor data using the activity classifier and the integrated feature vector.

6. The method of claim 1 , wherein determining, for each of the two or more sensors, the location data comprises detecting, for each of the two or more sensors, location data from a map-based feature embedding space that indicates a location of the corresponding sensor.

7. The method of claim 6 , wherein the map-based feature embedding space is a two-dimensional representation of (i) a physical location and (ii) one or more features of one or more sensors.

8. The method of claim 1 , wherein determining the classification of the activity within the building and the location of the activity using the integrated feature vector comprises providing, as input to a machine learning model using convolutional neural networks, the received sensor data.

9. The method of claim 8 , wherein the machine learning model implements a loss function for activity classification, representing a type of the activity.

10. The method of claim 9 , wherein the machine learning model implements a loss function for activity localization, representing the location of the activity.

11. The method of claim 1 , wherein the activity indicates two or more different actors.

12. The method of claim 1 , wherein the activity indicates two or more different locations.

13. The method of claim 1 , wherein the monitoring action comprises taking at least one of photo or video.

14. The method of claim 1 , wherein the monitoring action comprises activating a building automation system to perform an action.

15. A system comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor, the memory storing instructions which, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving, from two or more sensors, sensor data indicating activity within a building;
determining, for each of the two or more sensors and using the received sensor data, (i) an extracted feature vector for the activity within the building and (ii) location data;
labelling each of the extracted feature vectors with the location data;
generating, using the labelled extracted feature vectors, an integrated feature vector;
determining, using the integrated feature vector as input to an activity classifier, a classification of the activity within the building and a location of the activity; and
in response to determining the classification of the activity within the building and the location of the activity, performing a monitoring action.

16. The system of claim 15 , wherein the location data indicates a location of the sensor.

17. The system of claim 16 , wherein the integrated feature vector represents a composite of two or more different extracted feature vectors that each represent one or more features of a single sensor.

18. The system of claim 16 , wherein determining the classification of the activity within the building and the location of the activity comprises classifying the received sensor data using the activity classifier and the integrated feature vector.

19. The system of claim 16 , wherein determining, for each of the two or more sensors, the location data comprises detecting, for each of the two or more sensors, location data from a map-based feature embedding space that indicates a location of the corresponding sensor.

20. A computer-readable storage device storing instructions that when executed by one or more processors cause the one or more processors to perform operations comprising:
receiving, from two or more sensors, sensor data indicating activity within a building;
determining, for each of the two or more sensors and using the received sensor data, (i) an extracted feature vector for the activity within the building and (ii) location data;
labelling each of the extracted feature vectors with the location data;
generating, using the labelled extracted feature vectors, an integrated feature vector;
determining, using the integrated feature vector as input to an activity classifier, a classification of the activity within the building and a location of the activity; and
in response to determining the classification of the activity within the building and the location of the activity, performing a monitoring action.

Assignees

• Objectvideo Labs Llc
• Object Video Labs Llc

Inventors

• Lee Jangwon

Classifications

• G05B19/02 Primary

  electric · CPC title

• G05B2219/25011

  Domotique, I-O bus, home automation, building automation · CPC title

• G06F18/253 Primary

  of extracted features · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/08

  Learning methods · CPC title