Data-driven activity prediction

Patent metadata

Field	Value
Publication number	US-11551103-B2
Application number	US-201916583714-A
Country	US
Kind code	B2
Filing date	Sep 26, 2019
Priority date	Oct 16, 2015
Publication date	Jan 10, 2023
Grant date	Jan 10, 2023

Abstract

A physical environment is equipped with a plurality of sensors (e.g., motion sensors). As individuals perform various activities within the physical environment, sensor readings are received from one or more of the sensors. Based on the sensor readings, activities being performed by the individuals are recognized and the sensor data is labeled based on the recognized activities. Future activity occurrences are predicted based on the labeled sensor data. Activity prompts may be generated and/or facility automation may be performed for one or more future activity occurrences.

First claim

What is claimed is:

1. A method comprising:
for a cognitively-impaired individual living within a smart home environment, determining an activity of interest;
querying an activity prediction server for a predicted future time associated with the activity of interest;
receiving, from the activity prediction server, the predicted future time associated with the activity of interest;
comparing the predicted future time to a current time; and
when the current time is equal to or greater than the predicted future time, presenting an activity prompt associated with the activity of interest, wherein the activity prediction server is trained based on:
defining a number of labeled sensor event data based on sensor readings;
determining a number of local features of the labeled sensor event data, the number of local features being computed from a sensor event window; and
determining a number of context features associated with the labeled sensor event data, the context features capturing a number of activity predictions from a history window, wherein the context features are defined based on a set of the activity predictions and a plurality of the number of local features,
wherein training the activity prediction server comprises:
extracting the number of local features from the labeled sensor event data;
training a multi-output regression learner by:
training a first machine learning model for the activity of interest using the number of local features to create a first trained machine learning model;
predicting times for the activity of interest using the first trained machine learning model;
extracting context features using the number of local features and predicted times for the activity of interest; and
training a second machine learning model for the activity of interest using the context features to create a second trained machine learning model,
wherein the first machine learning model and the second machine learning model are trained based in part on a dataset aggregation (Dagger) algorithm enhanced with an aggregated imitation training.

2. The method of claim 1 , wherein the presentation of the activity prompt is based on user-defined preferences defining a relative time at which the activity prompt is to be presented.

3. The method of claim 1 , wherein the presentation of the activity prompt is based on user-defined preferences defining whether a visual indicator, audio indicator, tactile indicator, or combinations thereof are to be presented as part of the activity prompt.

4. The method of claim 1 , wherein querying the activity prediction server includes periodically querying the activity prediction server.

5. The method of claim 1 , wherein determining the activity of interest includes one or more of:
determining an activity specified in association with user-configured settings;
determining an activity for which no activity prompts are currently pending; or
determining an activity for which an activity prompt is currently pending and is schedule to be presented within a threshold time period.

6. The method of claim 1 , further including:
receiving user feedback indicating an action taken by a user in response to the activity prompt; and
determining an effectiveness of the activity prompt for the user based on the user feedback.

7. The method of claim 6 , wherein the user feedback includes an indication from the user that the user has performed the activity, the user is going to perform the activity, the user will perform the activity at a later time, or the user is choosing to ignore the activity.

8. The method of claim 6 , wherein the user feedback is used to improve an accuracy of a future activity occurrence prediction.

9. The method of claim 6 , further including:
sending the user feedback to the activity prediction server; and
with the activity prediction server, retraining an activity occurrence predictor based on the user feedback.

10. A method comprising:
for a cognitively-impaired individual living within a smart home environment:
training an activity recognition module of an activity prediction server based on a set of activity recognition training data to create a trained activity recognition module;
receiving sensor events from an environment; using the trained activity recognition module to label the sensor events with an activity class to generate labeled sensor event data;
training an activity occurrence predictor based, at least in part, on the labeled sensor event data;
using the activity occurrence predictor to predict a future occurrence time of an activity within the environment;
presenting, on a user interface, an activity prompt associated with the activity in response to a current time being equal to or greater than the predicted future occurrence time;
receiving user feedback via the user interface indicating an action taken by a user in response to the activity prompt; and
retraining the activity occurrence predictor based on the user feedback, wherein the activity prediction server is trained based on:
defining a number of labeled sensor event data based on sensor readings;
determining a number of local features of the labeled sensor event data, the number of local features being computed from a sensor event window; and
determining a number of context features associated with the labeled sensor event data, the number of context features capturing a number of activity predictions from a history window, wherein the number of context features are defined based on a set of the activity predictions and a plurality of the number of local features,
wherein training the activity prediction server comprises:
extracting the number of local features from the labeled sensor event data;
training a multi-output regression learner by:
training a first machine learning model for an activity of interest using the number of local features to create a first trained machine learning model;
predicting times for the activity of interest using the first trained machine learning model;
extracting the number of context features using the number of local features and predicted times for the activity of interest; and
training a second machine learning model for the activity of interest using the number of context features to create a second trained machine learning model,
wherein the first machine learning model and the second machine learning model are trained based in part on a dataset aggregation (Dagger) algorithm enhanced with an aggregated imitation training.

11. The method of claim 10 , wherein:
the set of activity recognition training data includes a plurality of sensor events, and
the sensor events are labeled with a corresponding activity class.

12. The method of claim 10 , wherein the presentation of the activity prompt is based on user-defined preferences defining a relative time at which the activity prompt is to be presented.

13. The method of claim 10 , wherein the user feedback includes an indication from the user that the user has performed the activity, the user is going to perform the activity, the user will perform the activity at a later time, or the user is choosing to ignore the activity.

14. The method of claim 13 , wherein retraining the activity occurrence predictor based on the user feedback includes using the user feedback to improve an accuracy of a future activity occurrence prediction.

15. A computing device comprising:
a processor;
a memory communicatively coupled to the processor; and
an activity prompting application stored in the memory and executed on the processor, to configure the computing device to:
deter

Assignees

• Univ Washington State

Inventors

• Cook Diane J
• Minor Bryan
• Doppa Janardhan Rao

Classifications

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/02 Primary

  Knowledge representation; Symbolic representation · CPC title