System and method for automatically discovering, characterizing, classifying and semi-automatically labeling animal behavior and quantitative phenotyping of behaviors in animals

Patent metadata

Field	Value
Publication number	US-11263444-B2
Application number	US-201816007924-A
Country	US
Kind code	B2
Filing date	Jun 13, 2018
Priority date	May 10, 2012
Publication date	Mar 1, 2022
Grant date	Mar 1, 2022

Abstract

A method for studying the behavior of an animal in an experimental area including stimulating the animal using a stimulus device; collecting data from the animal using a data collection device; analyzing the collected data; and developing a quantitative behavioral primitive from the analyzed data. A system for studying the behavior of an animal in an experimental area including a stimulus device for stimulating the animal; a data collection device for collecting data from the animal; a device for analyzing the collected data; and a device for developing a quantitative behavioral primitive from the analyzed data. A computer implemented method, a computer system and a nontransitory computer readable storage medium related to the same. Also, a method and apparatus for automatically discovering, characterizing and classifying the behavior of an animal in an experimental area. Further, use of a depth camera and/or a touch sensitive device related to the same.

First claim

We claim:

1. A method for automatically discovering, characterizing and classifying the behavior of an animal in an experimental area, comprising:
(a) using a 3D depth camera to obtain a video stream having a plurality of images of the animal, the images having both area and depth information;
(b) removing background noise from each of the plurality of images to generate processed images having light and dark areas, wherein the light areas comprise depth values above a threshold of a subtracted background and wherein the dark areas comprise depth values at or below the threshold of the subtracted background;
(c) determining contours of the light areas in the plurality of processed images;
(d) extracting at least one morphometric parameter from both area and depth image information within the contours to form a plurality of multi-dimensional data points, each data point representing the posture of the animal at a specific time, wherein the at least one morphometric parameter comprises a head direction of the animal;
(e) clustering the data points at each specific time to output a set of clusters that are segmented from each other so that each cluster represents animal behavior, wherein the step of clustering the data points at each specific time further comprises reducing covariance between data points scanned with a search algorithm in a plurality of sliding windows each having a different time duration, and clustering the data points with reduced covariance with a clustering method;
(f) assigning each cluster a label that represents animal behavior; and
(g) outputting the set of clusters and corresponding labels.

2. The method of claim 1 wherein step (b) comprises:
(b1) using the 3D depth camera to obtain a video stream having a plurality of baseline images without the animal present;
(b2) generating a median of the baseline images to form a baseline depth image;
(b3) subtracting the baseline depth image from each of the plurality of images obtained in step (a) to produce a plurality of difference images;
(b4) performing a median filtering operation on each difference image to generate a filtered difference image; and
(b5) removing image data that is less than a predetermined threshold from each of the plurality of filtered difference images to generate the processed images.

3. The method of claim 1 , wherein step (c) comprises determining contours of all light regions in each processed image with a contour detection algorithm and tracking each contour with a Kalman filter.

4. The method of claim 1 , wherein the label for each cluster is requested by a user interface after displaying video data representing each cluster.

5. The method of claim 1 , wherein the behavior comprises a quantitative behavior primitive.

6. The method of claim 1 , wherein step (b) further comprises:
(b1) using the 3D depth camera to obtain a video stream having a plurality of baseline images without the animal present;
(b2) generating a median of the baseline images to form a baseline depth image;
(b3) subtracting the baseline depth image from each of the plurality of images obtained in step (a) to produce a plurality of difference images;
(b4) performing an object detection operation, using a Haar cascade object detector, on each difference image to generate an object-detected difference image; and
(b5) removing image data that is less than a predetermined threshold from each of the plurality of object-detected difference images to generate the processed images.

7. An apparatus for automatically discovering, characterizing and classifying the behavior of an animal in an experimental area, comprising:
a 3D depth camera that generates a video stream having a plurality of images of the animal, the images having both area and depth information;
a data processing system having a processor and a memory containing program code which, when executed:
(a) removes background noise from each of the plurality of images to generate processed images having light and dark areas, wherein the light areas comprise depth values above a threshold of a subtracted background and wherein the dark areas comprise depth values at or below the threshold of the subtracted background;
(b) determines contours of the light areas in the plurality of processed images;
(c) extracts at least one morphometric parameter from both area and depth image information within the contours to form a plurality of multi-dimensional data points, each data point representing the posture of the animal at a specific time, wherein one of the at least one morphometric parameter comprises a head direction of the animal; and
(d) clusters the data points at each specific time to output a set of clusters that are segmented from each other so that each cluster represents animal behavior, wherein the data processing system is further configured to cluster the data points at each specific time by reducing covariance between data points scanned with a search algorithm in a plurality of sliding windows each having a different time duration;
(e) assigns each cluster a label that represents animal behavior; and
(f) outputs the set of clusters and corresponding labels.

8. The apparatus of claim 7 , wherein the 3D depth camera obtains a video stream having a plurality of baseline images without the animal present, and wherein step (a) comprises:
(a1) generating a median of the baseline images to form a baseline depth image;
(a2) subtracting the baseline depth image from each of the plurality of images to produce a plurality of difference images;
(a3) performing a median filtering operation on each difference image to generate a filtered difference image; and
(a4) removing image data that is less than a predetermined threshold from each of the plurality of filtered difference images to generate the processed images.

9. The apparatus of claim 7 , wherein the label for each cluster is requested after displaying video data representing each cluster.

10. The apparatus of claim 7 , wherein the behavior comprises a quantitative behavior primitive.

11. The apparatus of claim 7 , wherein step (b) comprises determining contours of all light regions in each processed image with a contour detection algorithm and tracking each contour with a Kalman filter.

12. The apparatus of claim 7 wherein in step (c) parameters extracted from area information with each contour include at least one of perimeter, surface area rotation angle and length.

13. The apparatus of claim 7 , wherein in step (c) parameters extracted from depth information with each contour include at least one of height, width, depth, velocity, spine curvature, and limb position.

14. The apparatus of claim 7 , wherein step (d) further comprises clustering the data points with reduced covariance with a clustering method.

15. The apparatus of claim 14 , wherein covariance between data points is reduced by reducing dimensionality of each data point.

16. The apparatus of claim 15 , wherein the dimensionality of each data point is reduced by applying at least one of principal components analysis, singular value decomposition, independent components analysis and locally linear embedding to the points.

17. The apparatus of claim 7 , wherein the 3D depth camera obtains a video stream having a plurality of baseline images without the animal present, and wherein step (a) further comprises:
(a1) generating a median of the baseline images to form a baseline depth image;
(a2) subtracting the baseline depth image from each of the plurality of images to produce a plurality of difference images;

Assignees

• Harvard College

Inventors

• Datta Sandeep Robert
• Wiltschko Alexander B

Classifications

• F04C2270/041

  Controlled or regulated · CPC title

• G06T7/0016

  involving temporal comparison · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• A01K67/00

  Rearing or breeding animals, not otherwise provided for; New or modified breeds of animals · CPC title

• G16B20/00

  ICT specially adapted for functional genomics or proteomics, e.g. genotype-phenotype associations · CPC title