Systems and methods for behavior detection using 3D tracking and machine learning

Patent metadata

Field	Value
Publication number	US-10121064-B2
Application number	US-201615183747-A
Country	US
Kind code	B2
Filing date	Jun 15, 2016
Priority date	Apr 16, 2015
Publication date	Nov 6, 2018
Grant date	Nov 6, 2018

Abstract

Systems and methods for performing behavioral detection using three-dimensional tracking and machine learning in accordance with various embodiments of the invention are disclosed. One embodiment of the invention involves a the classification application that directs a microprocessor to: identify at least a primary subject interacting with a secondary subject within a sequence of frames of image data including depth information; determine poses of the subjects; extract a set of parameters describing the poses and movement of at least the primary and secondary subjects; and detect a social behavior performed by at least the primary subject and involving at least the second subject using a classifier trained to discriminate between a plurality of social behaviors based upon the set of parameters describing poses and movement.

First claim

What is claimed is:

1. A behavioral classification system, comprising:
a microprocessor; and
memory containing a classification application;
wherein the classification application directs the microprocessor to:
identify at least a primary subject interacting with a secondary subject within a sequence of frames of image data comprising depth information;
determine poses for at least the primary subject and the secondary subject within a plurality of frames from the sequence of frames of image data;
extract a set of parameters describing the poses and movement of at least the primary and secondary subjects from the plurality of frames from the sequence of frames of image data; and
detect a social behavior performed by at least the primary subject and involving at least the secondary subject, wherein the primary subject occludes at least a portion of the secondary subject, using a classifier trained to discriminate between a plurality of social behaviors based upon the set of parameters describing poses and movement of a plurality of subjects extracted from a plurality of frames of image data comprising depth information.

2. The behavioral classification system of claim 1 , wherein the classifier is trained to discriminate between a plurality of social behaviors using a training data set comprising a plurality of sequences of frames of image data comprising depth information.

3. The behavioral classification system of claim 2 , wherein:
each sequence of frames of image data comprising depth information in the training data set is annotated using one of a predetermined set of a plurality of social behaviors; and
the classifier is trained to discriminate between behaviors within the predetermined set of a plurality of social behaviors.

4. The behavioral classification system of claim 2 , wherein:
the training of the classifier using the training data set automatically generates a set of a plurality of social behaviors observed in the training data set; and
the classifier is trained to discriminate between behaviors within the automatically generated set of a plurality of social behaviors observed in the training data set.

5. The behavioral classification system of claim 1 , wherein the classification application further directs the microprocessor to detect occurrence of modified social behavior in at least the primary subject resulting from administration of a pharmaceutical.

6. The behavioral classification system of claim 1 , wherein the classification application further directs the microprocessor to detect a behavioral phenotype associated with a genotype of the primary subject based upon detection of a pattern of social behaviors including the detected social behavior by a set of subjects including at least the primary subject that share the same genotype.

7. The behavioral classification system of claim 1 , wherein the primary and secondary subjects are rodents.

8. The behavioral classification system of claim 7 , wherein the plurality of behaviors comprise a plurality of behaviors selected from the group consisting of: attack, close inspection, mounting, chasing, social grooming, maternal behavior, paternal behavior, female receptivity, and social feeding.

9. The behavioral classification system of claim 7 , wherein the classification application further directs the microprocessor to detect occurrence of modified social behavior in at least the primary subject resulting from administration of a pharmaceutical.

10. The behavioral classification system of claim 7 , wherein the classification application further directs the microprocessor to detect a behavioral phenotype associated with a genotype of the primary subject based upon detection of a pattern of social behaviors including the detected social behavior by a set of subjects including at least the primary subject that share the same genotype.

11. The behavioral classification system of claim 1 , wherein the primary and secondary subjects are non-human primates.

12. The behavioral classification system of claim 11 , wherein the classification application further directs the microprocessor to detect occurrence of modified social behavior in at least the primary subject resulting from administration of a pharmaceutical.

13. The behavioral classification system of claim 11 , wherein the classification application further directs the microprocessor to detect a behavioral phenotype associated with a genotype of the primary subject based upon detection of a pattern of social behaviors including the detected social behavior by a set of subjects including at least the primary subject that share the same genotype.

14. The behavioral classification system of claim 1 , wherein the classification application directs the microprocessor to identify at least a primary subject interacting with a secondary subject within a sequence of frames of image data comprising depth information by:
performing background subtraction using a plurality of frames of image data; and
performing segmentation of at least a primary subject and a secondary subject.

15. The behavioral classification system of claim 14 , wherein the classification application further directs the microprocessor to identify at least a primary subject interacting with a secondary subject within a sequence of frames of image data comprising depth information based upon characteristic markings of primary and second subjects visible within frames of image data comprising video data in at least one color channel.

16. The behavioral classification system of claim 1 , wherein the classifier is selected from the group consisting of a support vector machine, adaptive boosting, and a random decision forest.

17. The behavioral classification system of claim 1 , wherein the image data further comprises video data in at least one color channel.

18. The behavioral classification system of claim 17 , further comprising:
a 3D imaging system;
wherein the classification application further directs the microprocessor to:
control the 3D imaging system to acquire the sequence of frames of image data comprising depth information and video image data in at least one color channel; and
store the sequence of frames of image data comprising depth information in memory.

19. The behavioral classification system of claim 18 , wherein the 3D imaging system is selected from the group consisting of:
a time of flight depth sensor and at least one camera;
a structured light depth sensor and at least one camera;
a LIDAR depth sensor and at least one camera;
a SONAR depth sensor and at least one camera;
a plurality of cameras in a multiview stereo configuration; and
a plurality of cameras in multiview stereo configuration and an illumination source that projects texture.

20. The behavioral classification system of claim 19 , wherein the 3D imaging system further comprises an additional camera.

21. The behavioral classification system of claim 19 , wherein the camera is selected from the group consisting of a monochrome camera, a Bayer camera, and a near-IR camera.

22. The behavioral classification system of claim 1 , wherein the classification application further directs the microprocessor to:
extract a set of parameters describing the poses and movement of at least the primary and secondary subjects from the plurality of frames from the sequence of frames of image data and from additional sensor data; and
the classifier is trained to discriminate between a plurality of social behavio

Assignees

• California Inst Of Techn

Inventors

• Hong Weizhe
• Anderson David J
• Perona Pietro

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title

• G06V10/40

  Extraction of image or video features · CPC title

• G16H30/40 Primary

  for processing medical images, e.g. editing · CPC title