Context-based detection and classification of actions

Patent metadata

Field	Value
Publication number	US-9305216-B1
Application number	US-201414570169-A
Country	US
Kind code	B1
Filing date	Dec 15, 2014
Priority date	Dec 15, 2014
Publication date	Apr 5, 2016
Grant date	Apr 5, 2016

Abstract

Actions or activities occurring within an environment may be detected, recognized and classified based on the presence or absence of objects within the environment, which may be recognized within imaging data. The states or changes in states of the objects may be tracked within the imaging data and associated with one or more actions or activities with various probabilities. By tracking the motion of objects, rather than the motion of humans or other actors, the detection and classification of actions or activities may be performed more quickly and efficiently, and may be used to determine the timing associated with one or more steps of such activities, as well as whether each of the steps of an activity has been satisfactorily performed and in an appropriate order.

First claim

What is claimed is:

1. A monitoring system comprising:
a plurality of imaging devices; and
a computing device in communication with at least one of the plurality of imaging devices,
wherein the computing device is configured to implement one or more services, and
wherein the one or more services are configured to:
cause a first set of imaging data to be captured by at least one of the plurality of imaging devices;
provide at least a portion of the first set of imaging data as an input to at least one first classifier, wherein the at least one first classifier is configured to recognize at least one object within the first set of imaging data;
recognize the at least one object within the imaging data based at least in part on an output from the at least one first classifier;
recognize a state of the at least one object based at least in part on the output from the at least one first classifier;
provide information regarding the at least one object and the state of the at least one object as an input to at least one second classifier, wherein the at least one second classifier is configured to associate at least one of the object or the state of the at least one object with one of a predetermined number of actions or activities;
identify at least one action or activity based at least in part on an output from the at least one second classifier; and
store information regarding the action or the activity in at least one data store.

2. The monitoring system of claim 1 , wherein the at least one first classifier is configured to recognize at least one of an edge, a contour, an outline, a color, a texture, a silhouette or a shape within the imaging data.

3. The monitoring system of claim 2 , wherein the at least one first classifier is configured to determine whether the at least one of the edge, the contour, the outline, the color, the texture, the silhouette or the shape within the imaging data is associated with the at least one object, and
wherein the at least one first classifier is one of
a trained support vector machine;
a Bayes classifier;
a neural network;
a Random Forest method; or
a deep learning method.

4. The monitoring system of claim 1 , wherein the at least one second classifier is configured to calculate a probability that the state of the at least one object is associated with the at least one of the predetermined number of actions or activities, and
wherein the one or more services are further configured to:
select the one of the predetermined number of actions or activities having a highest probability.

5. The monitoring system of claim 1 , wherein the predetermined number of actions or activities comprises at least one of:
receiving the at least one object;
changing a position of the at least one object;
inspecting the at least one object;
identifying a container for the at least one object;
placing the at least one object into the container; or
sealing the container with the at least one object therein.

6. A computer-implemented method comprising:
identifying information regarding an object within a scene of an environment based at least in part on imaging data captured from the environment;
identifying information regarding a state of the object within the scene of the environment based at least in part on the imaging data captured from the environment;
providing the information regarding the object and the information regarding the state of the object to at least one first classifier as inputs;
receiving at least one output from the at least one first classifier;
identifying at least one action or activity associated with the object or the state of the object based at least in part on the at least one output from the at least one first classifier; and
storing information regarding the at least one action or activity in at least one data store.

7. The computer-implemented method of claim 6 , wherein the at least one first classifier is configured to determine whether at least one of the object or the state of the object is associated with one of a plurality of predetermined actions or activities.

8. The computer-implemented method of claim 6 , wherein identifying the information regarding the state of the object within the scene of the environment further comprises:
tracking motion of the object within the imaging data,
wherein the information regarding the state of the object within the scene of the environment is identified based at least in part on the tracked motion of the object within the imaging data, and
wherein the at least one action or activity associated with the object or the state of the object is identified based at least in part on the tracked motion.

9. The computer-implemented method of claim 7 , wherein identifying the at least one action or activity associated with the object or the state of the object further comprises:
determining a probability that each of the plurality of predetermined actions or activities is associated with at least one of the object or the state of the object based at least in part on the at least one output from the at least one first classifier; and
selecting the one of the plurality of predetermined actions or activities having a highest probability,
wherein the at least one action or activity is associated with the selected one of the predetermined actions or activities.

10. The computer-implemented method of claim 9 , further comprising:
determining a time at which the selected one of the predetermined actions or activities has the highest probability; and
associating the time with a performance of the selected one of the predetermined actions or activities having the highest probability.

11. The computer-implemented method of claim 10 , further comprising:
determining times at which the probability that each of at least two of the predetermined actions or activities is associated with at least one of the object or the state of the object exceed a predetermined threshold; and
associating each of the times with a performance of a respective one of the at least two of the plurality of predetermined actions or activities.

12. The computer-implemented method of claim 11 , further comprising:
identifying a productivity standard associated with the at least two of the plurality of predetermined actions or activities; and
determining whether each of the at least two of the plurality of predetermined actions or activities complies with the productivity standard based at least in part on the times.

13. The computer-implemented method of claim 11 , further comprising:
identifying an established procedure associated with the object or the state of the object; and
determining whether the established procedure associated with the object or the state of the object comprises the at least two of the plurality of predetermined actions or activities.

14. The computer-implemented method of claim 13 , wherein determining whether the established procedure associated with the object or the state of the object comprises the at least two of the plurality of predetermined actions or activities comprises:
determining whether the established procedure associated with the object or the state of the object consists of the at least two of the plurality of predetermined actions or activities.

15. The computer-implemented method of claim 6 , wherein identifying the information regarding the object within the scene of the environment based at least in part on the imaging data captured from the environment comprises:
detecting the at least one object in the imaging data using at least one computer processor.

Assignees

• Amazon Tech Inc

Inventors

• Mishra Pragyana K

Classifications

• G06V20/52 Primary

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06F16/55

  Clustering; Classification · CPC title

• G06K9/4604

  Physics · mapped topic