Video processing for motor task analysis

Patent metadata

Field	Value
Publication number	US-10083233-B2
Application number	US-201414536660-A
Country	US
Kind code	B2
Filing date	Nov 9, 2014
Priority date	Sep 9, 2014
Publication date	Sep 25, 2018
Grant date	Sep 25, 2018

Abstract

Video processing for motor task analysis is described. In various examples, a video of at least part of a person or animal carrying out a motor task, such as placing the forefinger on the nose, is input to a trained machine learning system to classify the motor task into one of a plurality of classes. In an example, motion descriptors such as optical flow are computed from pairs of frames of the video and the motion descriptors are input to the machine learning system. For example, during training the machine learning system identifies time-dependent and/or location-dependent acceleration or velocity features which discriminate between the classes of the motor task. In examples, the trained machine learning system computes, from the motion descriptors, the location dependent acceleration or velocity features which it has learned as being good discriminators. In various examples, a feature is computed using sub-volumes of the video.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
receiving a video depicting at least part of a user performing a motor task;
redacting features from the video which may identify the user;
inputting the video to a trained machine learning system, the trained machine learning system having been trained to detect multiple sequences of location-dependent local motion features of videos which discriminate among a plurality of classes of the motor task; and
receiving, from the trained machine learning system, a label identifying a motor task class of the plurality of classes, the plurality of classes including
an indication of a performance level of the motor task based on at least one of the multiple sequences of location-dependent local motion features in the video detected by the trained machine learning system.

2. A method as claim 1 recites, wherein the local motion features comprise velocity or acceleration features.

3. A method as claim 2 recites further comprising calculating the acceleration features by taking into account frequency of change of direction of rate of change of optical flow values of a sub-volume of the video.

4. A method as claim 3 recites further comprising disregarding changes of direction of the rate of change of the optical flow values, where a magnitude of the optical flow values is below a threshold.

5. A method as claim 1 recites further comprising calculating motion descriptors for pairs of frames of the video, and wherein inputting the video to the trained machine learning system comprises inputting the motion descriptors.

6. A method as claim 5 recites, wherein the motion descriptors include optical flow values.

7. A method as claim 5 recites further comprising, at the machine learning system, calculating the local motion features using a plurality of the motion descriptors.

8. A method as claim 5 recites, wherein the trained machine learning system finds the local motion features by performing a calculation based at least in part on the motion descriptors in at least one sub-volume of the video.

9. A method as claim 5 recites further comprising calculating the local motion features by taking into account differences between motion descriptors in sub-volumes of the video.

10. A method as claim 1 recites further comprising pre-processing the video prior to inputting the video to the trained machine learning system, at least by one or more of scaling, centering, or carrying out foreground extraction.

11. A method as claim 1 recites, wherein:
the video is of any length, and
the trained machine learning system calculates the local motion features based at least in part on the length of the video.

12. A method as claim 1 recites further comprising training the machine learning system using videos of people performing the motor task, where the videos are labeled with labels indicating which of a plurality of possible classes the motor task belongs to, and where the videos are of different lengths.

13. A method as claim 1 recites further comprising inputting the video to a trained machine learning system comprising one or more of: a random decision forest, a jungle of directed acyclic graphs, or an ensemble of support vector machines.

14. A method as claim 1 recites further comprising inputting the video to a trained machine learning system comprising an ensemble of support vector machines, a support vector machine of the ensemble including a split node of a binary decision tree.

15. A method as claim 1 recites further comprising inputting the video to a trained machine learning system comprising an ensemble of support vector machines, individual ones of the support vector machines having been trained using fixed length feature descriptors comprising randomized location-dependent local motion features computed from labeled training videos.

16. A computer storage media storing instructions comprising:
instructions to receive a video depicting at least part of a person or animal performing a motor task;
instructions to input the video to a trained machine learning system, having been trained to detect multiple sequences of location-dependent local acceleration features of videos which discriminate among a plurality of classes of the motor task, the local acceleration features calculated by taking into account frequency of change of direction of rate of change of optical flow values of a sub-volume of the video; and
instructions to receive, from the trained machine learning system a label identifying a motor task class of the plurality of classes, wherein the plurality of classes includes
an indication of a performance level of the motor task based on at least one of the multiple sequences of location-dependent local motion features in the video detected by the trained machine learning system.

17. A motor-task classifier comprising:
a trained machine learning system, having been trained to detect multiple sequences of location-dependent local motion features of videos which discriminate among a plurality of classes of a motor task, the training comprising optimizing a criteria based at least in part on a duration of a sub-volume; and
a processor arranged to:
compute motion descriptors from a video;
input the motion descriptors to the trained machine learning system; and
receive, from the trained machine learning system, a label identifying a motor task class of the plurality of classes, wherein the plurality of classes includes
an indication of a performance level of the motor task based on at least one of the multiple sequences of location-dependent local motion features in the video detected by the trained machine learning system.

18. A motor-task classifier as claim 17 recites, wherein the local motion features include velocity or acceleration features.

19. A motor-task classifier as claim 17 recites, wherein computing motion descriptors from the video includes calculating motion descriptors for pairs of frames of the video.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Kontschieder Peter
• Dorn Jonas
• Zikic Darko
• Criminisi Antonio

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N5/00

  Computing arrangements using knowledge-based models · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G16H50/20

  for computer-aided diagnosis, e.g. based on medical expert systems · CPC title